{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross-entropy (CE) Method Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CE method test 1\n",
    "The distribution in CE method directly fits the f-value, in this approach, the f-value has to be the directly fitted by the cem, it cannot depend on other parameters. But it works! How to prove the convergence?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('true min: ', -2.921757706470594)\n",
      "[-3.05239919 -3.05239919 -3.05239919 -3.05239919 -3.05239919 -3.05239919\n",
      " -3.05239919 -3.05239919 -3.05239919 -3.05239919]\n"
     ]
    }
   ],
   "source": [
    "# cem test 1\n",
    "# cem directly fits the f-value, in this approach, the f has to be the direct\n",
    "# values fitted by the cem, it cannot depend on other parameters. But it works!\n",
    "import numpy as np\n",
    "N=100 # number of samples\n",
    "ratio=0.1 # ratio of selection\n",
    "num_itr=50\n",
    "samples = np.random.normal(0,1,N)  # f-value: from a normal distribution\n",
    "print('true min: ', np.min(samples))\n",
    "for i in range(num_itr):\n",
    "    # sample and select top 10% (max [-a:]or min [:a])\n",
    "    selected_samples = samples[np.array(samples).argsort()[:int(N*ratio)]]\n",
    "#     selected_samples = samples[np.array(samples).argsort()[-int(N*ratio):]]\n",
    "    m=np.mean(selected_samples)\n",
    "    std=np.std(selected_samples)\n",
    "    # fit the gaussian\n",
    "    samples=np.random.normal(m, std, N)\n",
    "print(selected_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Results from above: \n",
    "* After CE iterations, the values close to the ture minimal of original samples or the f-distribution are finally sampled, which testifies directly fitting original f-value could work in some situations.\n",
    "* And the standard deviation in the fitting Gaussian distribution of CE method decreases very fast actually. \n",
    "* The final sampled values are actually still not the true optimal value of the original distribution, just close to it. CE method actually cannot be accurate, because it always depends on the number of samples used for evaluation. The larger the number, the more accurate it is.　This method could be accurate to the infinite large limit number of samples, under the law of large numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CE method test 2\n",
    "The distribution of CE method fits the x of f(x). In this approach, the x can be parameters of \n",
    "the function f, therefore f can be dependent on other variables z like f(x,z). It works \n",
    "as long as there exists minimum of the function f."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('true min: ', 8.241945158007902e-05)\n",
      "[4.55442613e-115 1.55757921e-113 3.13071712e-113 3.32578702e-113\n",
      " 4.62277571e-113 5.19920956e-113 5.30794955e-113 5.45446403e-113\n",
      " 5.59609879e-113 7.94580383e-113]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8XGed7/HPT6Nm2eqSbVnFPS5xi+3YjkkCxIlTCHE2kMUsEO8SNgTCLmUvu8mF14YLu3fhsnspd2khCQQIKYSEBEhwOiZxXOTeLblKbpKtYsmy6jz3jzkygyXZsqSZMyN936+XrJnnPDPz09F4vjrPc4o55xAREQmX4HcBIiISexQOIiLShcJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLhL9LqCv8vLy3Lhx4/wuQ0QkbmzYsOGkcy6/N33jNhzGjRtHaWmp32WIiMQNMzvU274aVhIRkS4UDiIi0oXCQUREulA4iIhIFwoHERHpQuEgIiJdKBxERKSLIRUOre1BfvDmPlbtrfa7FBGRS/bqzhM8tGof0bi885AKh6SA8dCqffx+6zG/SxERuWTPbTrCz9ccwswi/lpDKhzMjBmFmWw7Uu93KSIil2zbkXpmFmZG5bUuGg5m9qiZVZnZ9rC2HDN7xczKvO/ZXruZ2XfNrNzMtprZ3LDHrPD6l5nZirD2eWa2zXvMdy3CkTirKJO9JxpobuuI5MuIiAyouqZWDtc0MbMwKyqv15sth58CN53Xdj/wmnNuMvCadx/gZmCy93UP8AMIhQnwILAQWAA82BkoXp97wh53/msNqJmFmbQHHbuPN0TyZUREBtT2I6cBYmfLwTm3Cqg5r3kZ8Jh3+zHg9rD2n7mQNUCWmRUANwKvOOdqnHO1wCvATd6yDOfcOy40w/KzsOeKiJlFodTdVlkXyZcRERlQW4+EPrNiJhx6MMo5dwzA+z7Say8EKsL6VXptF2qv7KY9YsZkppIzPFnzDiISV7YfqackJ43MtKSovN5AT0h3N1/g+tDe/ZOb3WNmpWZWWl3dt91RzYyZhZlsrVQ4iEj82FpZz8yi6Gw1QN/D4YQ3JIT3vcprrwSKw/oVAUcv0l7UTXu3nHMPOefmO+fm5+f36noV3ZpZmElZVaMmpUUkLtSeaaWy9mzUhpSg7+HwAtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0UuBld6yBjNb5O2ldFfYc0XMzKJMOoKOncdOR/qlRET6rXMYfFYshYOZPQG8A0wxs0ozuxv4OnCDmZUBN3j3AV4E9gPlwI+BTwM452qArwHrva+vem0AnwIe9h6zD3hpYH60nnWm73bNO4hIHOgMh8ujGA4XvUyoc+7DPSxa0k1fB9zXw/M8CjzaTXspMONidQykgsxU8kYka95BROLCtsp6xuWmkTksOpPRMMSOkO7UeaS0thxEJB5sO1LPjChuNcAQDQcIjd3tPdHA2VZNSotI7DrV2MKRurPMiuKeSjCEw2FGYSZBBzuPaetBRGJX53yDthyiZNa5I6UVDiISu7YrHKJrVEYK+ekpmpQWkZi2tbKe8XnDyUiN3mQ0DOFwMDNmF2WyWedYEpEY5Zxjc0Uds6M83wBDOBwA5hRnsb/6DPVNbX6XIiLSxbH6ZqoaWphTHJ3TdIcb4uEQOmv4Fm09iEgM2lIR+myaU5J9kZ4Db0iHw6ziTMz+/AsQEYklmyvqSA4kMK0gPeqvPaTDISM1iYn5I9iscBCRGLSpoo5pYzJISQxE/bWHdDhAaN5hc0UdoTN/iIjEhvaOINsq67nCh/kGUDgwuziLU97pcEVEYkVZVSNn2zp8mYwGhcO5VN6koSURiSGdw90KB59MGZ1OSmKCJqVFJKZsPlxHVloSY3PTfHn9IR8OSYEEZhZmalJaRGJK6OC3LELXQYu+IR8OENps236knraOoN+liIjQ2NLO3qoG34aUQOEAhCalW9qD7D7W4HcpIiJsq6zHOZhTonDwVWc6b66o9bkSEZE/T0bPLlI4+Kooexh5I5K1x5KIxITNFbWMzU0jZ3iybzUoHAidofWKkmw2HVY4iIi/nHNsPFzn28FvnRQOnnljszlw8gynGlv8LkVEhrDK2rNUN7Qwb1yOr3UoHDzzxobOerhRWw8i4qPSQzUAzPPhTKzhFA6emYWZJAXs3C9GRMQPGw7VMiIlkSmjo38m1nAKB09qUoAZhZlsPKQ9lkTEPxsO1XFFSRaBBH8OfuukcAgzrySbLZX1tLbrYDgRib6G5jb2HD/NXJ+HlEDh8Bfmjc2mtT3IjqP1fpciIkPQ5oo6gg7mj1M4xJTOSekNGloSER9sOFSLmX9nYg2ncAgzMiOV4pxhCgcR8cWGQ7VMGZVOemqS36UoHM43rySb0kO1ujKciERVR9Cx6XBdTAwpgcKhi3ljs6luaNGV4UQkqvaeaKCxpf3c8Lbf+hUOZvZ5M9thZtvN7AkzSzWz8Wa21szKzOwpM0v2+qZ498u95ePCnucBr32Pmd3Yvx+pf+aNDR2VqKElEYmmUu8zZ16Jv0dGd+pzOJhZIfCPwHzn3AwgACwHvgF8yzk3GagF7vYecjdQ65ybBHzL64eZTfcedzlwE/B9Mwv0ta7+mjI6neHJAYWDiETVxkO15KenUJwzzO9SgP4PKyUCw8wsEUgDjgHXAc94yx8DbvduL/Pu4y1fYqFLHC0DnnTOtTjnDgDlwIJ+1tVngYTQSfjWH9SR0iISPesP1jCvJNu3K7+dr8/h4Jw7AvwncJhQKNQDG4A651y7160SKPRuFwIV3mPbvf654e3dPOYvmNk9ZlZqZqXV1dV9Lf2iFozPYc+JBuqaWiP2GiIinY7UnaWy9iwLxsfGkBL0b1gpm9Bf/eOBMcBw4OZuunbu9tNdHLoLtHdtdO4h59x859z8/Pz8Sy+6lxaMz8E5KD2ooSURibz1B0IjFYMiHIDrgQPOuWrnXBvwLLAYyPKGmQCKgKPe7UqgGMBbngnUhLd38xhfzCnOIjmQwDoNLYlIFKw9UEN6aiLTCjL8LuWc/oTDYWCRmaV5cwdLgJ3AG8AHvT4rgOe92y949/GWv+5CBxO8ACz39mYaD0wG1vWjrn5LTQowpziLtQcUDiISeesOnOLKcTm+n2wvXH/mHNYSmljeCGzznush4F+AL5hZOaE5hUe8hzwC5HrtXwDu955nB/A0oWD5A3Cfc66jr3UNlAXjc9h+pJ4zLe0X7ywi0kcnG1vYV30mpoaUILS3UZ855x4EHjyveT/d7G3knGsG7uzhef4d+Pf+1DLQFozP4b/fKGfj4VqumRy5+Q0RGdpicb4BdIR0j+aOzSaQYKzT0JKIRNDaAzUMSwowY0ym36X8BYVDD0akJDJjTAZr9yscRCRy1h6oYe7YLJITY+vjOLaqiTELJ+SyuaKO5jbfp0BEZBCqb2pj9/HTLByf63cpXSgcLmDBuBxaO4JsqajzuxQRGYRKD9XgXOzNN4DC4YKuHJeDGZp3EJGIWHeghuRAQkxc3Od8CocLyExLYsqodB3vICIRsfZADbOLM0lN8u1coz1SOFzEogm5lB6qobU96HcpIjKINDS3se1IfUzON4DC4aIWT8yluS3IZs07iMgAWn+who6gY/EkhUNcWjghlwSDt8tP+l2KiAwib5efIjkxgbklsXHlt/MpHC4ic1gSMwozeWffKb9LEZFBZPW+U8wfmx2T8w2gcOiVxRPz2FRRS1OrzrMkIv1Xc6aVXcdO865JeX6X0iOFQy8snphLW4djva7vICIDoHMk4qqJsTnfAAqHXpk/LpukgLF6n+YdRKT/Vu87yYiURGYVxtb5lMIpHHohLTmRK0qyWV2ueQcR6b/V+06xcHwOiYHY/QiO3cpizOKJuWw/Wk99U5vfpYhIHDtad5YDJ8/E9JASKBx6bfHEPJyDNQe09SAifdc537B4YuxORoPCodfmFGcxLCnAah3vICL98Pa+k+QMT2bq6HS/S7kghUMvJScmcOX4HN7W8Q4i0kfOOd7Zd4qrJuSSEEPXi+6OwuESXD0pl/KqRo7Vn/W7FBGJQ/uqGzlW3xzTxzd0UjhcgmsvC11L+k97NbQkIpfuj95nx7WXKRwGlSmj0hmVkcIfy6r9LkVE4tCqvdVMyB9OUXaa36VclMLhEpgZ10zO562yk3QEnd/liEgcaW7rYO2BU1w7Od/vUnpF4XCJrr0sn/qzbWyt1Cm8RaT31h+sobktyLsvUzgMSldPysMMVmneQUQuwaq91SQHElg4IfauF90dhcMlyhmezMzCTFZp3kFELsGqvSeZPy6btOREv0vpFYVDH1w7OZ/NFXXUn9WpNETk4o7XN7PnRMO5PR7jgcKhD669LJ+OoNPR0iLSK50jDfEyGQ0Khz65oiSLESmJrCpTOIjIxf2p7CT56SlMK4jtU2aEUzj0QVIggcUTc1m1txrntEuriPSsI+h4q6yaaybnYRbbp8wI169wMLMsM3vGzHab2S4zu8rMcszsFTMr875ne33NzL5rZuVmttXM5oY9zwqvf5mZrejvDxUN75kykiN1ZymvavS7FBGJYVsq66htaoubXVg79XfL4TvAH5xzU4HZwC7gfuA159xk4DXvPsDNwGTv6x7gBwBmlgM8CCwEFgAPdgZKLLtu6kgAXttd5XMlIhLLXt9VRSDBeM9lI/0u5ZL0ORzMLAO4FngEwDnX6pyrA5YBj3ndHgNu924vA37mQtYAWWZWANwIvOKcq3HO1QKvADf1ta5oGZ2ZyuVjMnh9l8JBRHr22u4q5o/NJjMtye9SLkl/thwmANXAT8xsk5k9bGbDgVHOuWMA3vfOuCwEKsIeX+m19dQe866bOpLSQzXUNbX6XYqIxKCjdWfZdez0uZGGeNKfcEgE5gI/cM5dAZzhz0NI3eluJsZdoL3rE5jdY2alZlZaXe3/QWjXTR1J0MEf9/pfi4jEnte9Yecl04ZWOFQClc65td79ZwiFxQlvuAjve1VY/+KwxxcBRy/Q3oVz7iHn3Hzn3Pz8fP8nd2YXZZE7PJnXNLQkIt14fXcVJTlpTMwf4Xcpl6zP4eCcOw5UmNkUr2kJsBN4Aejc42gF8Lx3+wXgLm+vpUVAvTfstBJYambZ3kT0Uq8t5iUkGO+dOpI391TR3hH0uxwRiSFnWzt4u/wk100dGVe7sHbq70k+/gF43MySgf3A3xEKnKfN7G7gMHCn1/dF4BagHGjy+uKcqzGzrwHrvX5fdc7V9LOuqFkydSTPbKhkw6FaFk7I9bscEYkRq/edpKU9GJdDStDPcHDObQbmd7NoSTd9HXBfD8/zKPBof2rxy9WT80gKGK/vrlI4iMg5r+2uYnhygAXj4+MsrOfTEdL9lJ6axMLxuecmnkREnHO8sbuKaybnk5IY8LucPlE4DIDrpo6krKqRgyfP+F2KiMSAHUdPc6y+mevidEgJFA4D4obpowBYueO4z5WISCxYueM4CRaak4xXCocBUJyTxozCDIWDiAChcFgwPofcESl+l9JnCocBcuP00Ww8XMeJ081+lyIiPtpf3cjeE43cePlov0vpF4XDALlxRuiN8PLOEz5XIiJ+Wrkj9BmwVOEgAJNHjmBC3nBe1tCSyJC2csdxZhVlUpg1zO9S+kXhMEDMjKWXj+adfaeob9K1pUWGouP1zWyuqIv7ISVQOAyoGy8fRXvQ8dpuDS2JDEUv7wyNHNx4+SifK+k/hcMAml2UxeiMVO21JDJErdxxnIn5w5k0Mn6uFd0ThcMASkgwll4+ij/uraaptd3vckQkiuqaWlmzv2ZQDCmBwmHA3TRjNM1tQd7YrWs8iAwlK3ccpyPouHlGgd+lDAiFwwBbOD6X/PQUfrul20tSiMgg9dstxxibGzogdjBQOAywQIJxy4zRvLGnisYWDS2JDAUnG1tYve8kt84qiMtrN3RH4RAB7589hpb2IK/qgDiRIeGl7ccJutD//cFC4RABc0uyKchM1dCSyBDx2y1HmTRyBFNGxf9eSp0UDhGQkGC8b2YBq8qqdUCcyCB34nQz6w/WDKohJVA4RMz7Z4+hrcOxcqeOeRAZzH6/9RjOwa2zBs+QEigcImZWUSYlOWkaWhIZ5H679SjTCjKYNHKE36UMKIVDhJgZ75tVwOp9pzjV2OJ3OSISAZW1TWw6XMetswbHsQ3hFA4RdNvsMXQEHb/beszvUkQkAp7fHBoZeP8gG1IChUNETSvIYFpBBs9urPS7FBEZYM45fr2xkgXjcijJTfO7nAGncIiwD8wtZEtlPeVVDX6XIiIDaEtlPfurz3DH3EK/S4kIhUOE3TZnDAkGz2484ncpIjKAnt1YSXJiArcMwvkGUDhE3Mj0VK69LJ/nNh0hGHR+lyMiA6C1PcgLW46ydPooMlKT/C4nIhQOUXDH3CKO1TezZv8pv0sRkQHwxp4q6pra+MDcIr9LiRiFQxQsnT6K9JREfq2hJZFB4dmNleSNSOGayXl+lxIxCocoSE0K8L5ZBby0/ZguAiQS52rPtPL67ipunzOGxMDg/QgdvD9ZjLljbhFNrR28uE2n0xCJZ89vPkJbh+OvBuleSp36HQ5mFjCzTWb2O+/+eDNba2ZlZvaUmSV77Sne/XJv+biw53jAa99jZjf2t6ZYdOW4bCbkDefJdYf9LkVE+sg5x5PrK5hZmMnlYzL9LieiBmLL4bPArrD73wC+5ZybDNQCd3vtdwO1zrlJwLe8fpjZdGA5cDlwE/B9MwsMQF0xxcxYvqCY0kO1lJ3QMQ8i8WhzRR27jzfw4QUlfpcScf0KBzMrAt4HPOzdN+A64Bmvy2PA7d7tZd59vOVLvP7LgCedcy3OuQNAObCgP3XFqg/MLSIpYDyxrsLvUkSkD55Yd5i05AC3zRl8p8s4X3+3HL4N/DMQ9O7nAnXOuc5Z10qgc2CuEKgA8JbXe/3PtXfzmEEld0QKS6eP5tlNlTS3dfhdjohcgobmNn675RjvnzWGESmJfpcTcX0OBzO7Fahyzm0Ib+6mq7vIsgs95vzXvMfMSs2stLq6+pLqjRXLFxRT19TGyh2amBaJJy9sOcrZtg6WLyj2u5So6M+Ww7uA28zsIPAkoeGkbwNZZtYZq0VA5wUNKoFiAG95JlAT3t7NY/6Cc+4h59x859z8/Pz8fpTun3dNzKM4ZxhPamhJJK48ua6CqaPTmVOc5XcpUdHncHDOPeCcK3LOjSM0ofy6c+4jwBvAB71uK4DnvdsvePfxlr/unHNe+3Jvb6bxwGRgXV/rinUJCcbyK0t4Z/8pDpw843c5ItIL24/Us+1IPR9eUDKoLgV6IZE4zuFfgC+YWTmhOYVHvPZHgFyv/QvA/QDOuR3A08BO4A/Afc65QT0gf+e8IhITjMfXHPK7FBHphV+sOURqUgK3zxmU06HdGpBZFefcm8Cb3u39dLO3kXOuGbizh8f/O/DvA1FLPBiZkcrNMwt4qrSCz99wGcOHwOSWSLyqPdPKc5uOcMfcIjLTBudJ9rqjI6R98reLx9LQ3M5zm3S+JZFY9lRpBS3tQVYsHut3KVGlcPDJ3JJsZhZm8tjqg4SmXkQk1rR3BPn5O4e4akIuU0dn+F1OVCkcfGJmrFg8jrKqRlbv06m8RWLRq7uqOFJ3lhWLx/ldStQpHHx066wCcoYn89PVB/0uRUS68djqgxRmDeP6aSP9LiXqFA4+Sk0K8DcLSnh11wkqapr8LkdEwuw+fpp39p/iY1eNHdSn5u7J0PuJY8xHFpUQMOMnbx/0uxQRCfPoWwdISUzgQ/OHxhHR51M4+Kwgcxi3zR7Dk+sPU9fU6nc5IgKcON3Mc5uO8KEri8kenux3Ob5QOMSAe949gabWDn6hg+JEYsKjbx+gI+j4xNUT/C7FNwqHGDB1dAbvmZLPT94+qLO1ivjsdHMbv1xzmFtmFlCSm+Z3Ob5ROMSIe989kVNnWnlmQ6XfpYgMaU+sPUxDSzv3vnui36X4SuEQIxaOz2F2cRY//tN+OoI6KE7EDy3tHTzy1gGunpTHjMLBfRnQi1E4xAgz41PvnsChU028tP2Y3+WIDEm/2XSEqoYWPvnuoTvX0EnhEENumD6aCfnD+e/Xywlq60Ekqto7gnz/zX3MKMzg6kl5fpfjO4VDDAkkGJ9dMpndxxt0pTiRKHtu0xEOnWric0suGzLXbLgQhUOMuXXWGCbkD+c7r5Vp60EkSto6gvy/18uZUZjBkiF4qozuKBxiTPjWwx+09SASFc9tOsLhGm01hFM4xKBzWw+vautBJNJCWw1lzCzM1FZDGIVDDOrcethzooGXtmvrQSSSntt4hIqas3zu+snaagijcIhRt84aw6SRI/ivV/bQ3hH0uxyRQam5rYNvv7qXWUWZXDdVWw3hFA4xKpBg/PONU9hffYanSiv8LkdkUHps9UGO1jdz/81TtdVwHoVDDLth+ijmj83mW6+Ucaal3e9yRAaVuqZWvvdGOe+Zks/iiTqu4XwKhxhmZjxwyzRONrbw8J8O+F2OyKDy/Tf30dDSzv03T/W7lJikcIhx88Zmc/OM0Ty0ah/VDS1+lyMyKFTWNvHTtw/ywblFTB2d4Xc5MUnhEAe+eOMUmtuDfPvVvX6XIjIo/OfKPZjBF5Ze5ncpMUvhEAcm5I/gY4vG8sS6w+w4Wu93OSJxbf3BGn6z+Sh/f80ECjKH+V1OzFI4xInP33AZ2WnJPPj8DpzTgXEifdERdPzr8zsYk5nKp987tK/XcDEKhziROSyJf7lpKqWHavnN5iN+lyMSl3659hC7jp3my7dOJy050e9yYprCIY58cF4Rs4uz+N8v7qahuc3vckTiyqnGFr65cg+LJ+Zy84zRfpcT8xQOcSQhwfjqbZdzsrGF77xa5nc5InHlmyv30NTawf+67XId8NYLCoc4M7s4i+VXlvDo2wfYWlnndzkicWHN/lM8ub6Cv3vXOCaPSve7nLjQ53Aws2Ize8PMdpnZDjP7rNeeY2avmFmZ9z3bazcz+66ZlZvZVjObG/ZcK7z+ZWa2ov8/1uB2/81TyRuRwj8/s5U2nXdJ5IKa2zq4/9dbKclJ4ws3TPG7nLjRny2HduCfnHPTgEXAfWY2HbgfeM05Nxl4zbsPcDMw2fu6B/gBhMIEeBBYCCwAHuwMFOle5rAk/u32Gew+3sCP/rjP73JEYtq3Xt3LwVNNfP2OmQxLDvhdTtzoczg454455zZ6txuAXUAhsAx4zOv2GHC7d3sZ8DMXsgbIMrMC4EbgFedcjXOuFngFuKmvdQ0VSy8fzftmFfDd18opr2rwuxyRmLStsp4fr9rP8iuLWazrQl+SAZlzMLNxwBXAWmCUc+4YhAIE6DwPbiEQfnrRSq+tp3a5iK+8/3LSUgJ88ZmtOq23yHla2jv44jNbyBuRwgO3TPO7nLjT73AwsxHAr4HPOedOX6hrN23uAu3dvdY9ZlZqZqXV1dWXXuwgk5+ewleXzWDT4Tq+94aGl0TC/efKPew+3sB/3DGTzGFJfpcTd/oVDmaWRCgYHnfOPes1n/CGi/C+V3ntlUBx2MOLgKMXaO/COfeQc26+c25+fn5+f0ofNG6bPYbb54zhu6+XsfFwrd/liMSEt8pO8uM/HeCji0pYMm2U3+XEpf7srWTAI8Au59z/DVv0AtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0Uu9Numlr94+g9EZqXzuyc006roPMsTVnmnln361mYn5w/nSLdP9Lidu9WfL4V3Ax4DrzGyz93UL8HXgBjMrA27w7gO8COwHyoEfA58GcM7VAF8D1ntfX/XapJcyUpP49vI5VNY28ZUXdvhdjohvnHM88Ow2as608p3lV2jvpH7o88lFnHNv0f18AcCSbvo74L4enutR4NG+1iJw5bgcPvPeSXz39XIWjMvhr68svviDRAaZR98+yB92HOeBm6cyozDT73Limo6QHkQ+e/1lXD0pjy8/v53tR3Rqbxla1h+s4T9e3MUN00dxz7UT/C4n7ikcBpFAgvGd5XPIG57Mvb/YQF1Tq98liURFVUMz9z2+kaLsYfzXX8/WuZMGgMJhkMkdkcL3PjKXE6eb+dxTm+kI6toPMri1tgf5zC83cbq5jR98dB4ZqdptdSAoHAahK0qy+cptl/Pmnmr+7fc7/S5HJGKcc3zpuW2sO1DDNz4wi2kFuh70QNHVLgapjywcy76qMzz69gHG5w3nrqvG+V2SyID7/pv7+NWGSv5xyWSWzdGJFQaSwmEQ+9L7pnG4JrR7a3F2Gu+dOvLiDxKJE7/bepRvrtzDsjlj+Pz1k/0uZ9DRsNIg1jlBPa0gg/t+uZFNOoJaBol39p3iC09vYf7YbL7xgVmagI4AhcMgNzwlkUf/9kryRqTwtz9Zz+7jFzr9lUjs21xRxyceW8/YnDQeums+qUk60C0SFA5DwKiMVB7/xEKGJQX46MPrOHDyjN8lifTJ7uOnWfHoOnJHpPCLTywkZ3iy3yUNWgqHIaI4J41ffGIBQef46MNrOXyqye+SRC5JeVUDH3tkHalJCTz+iYWMykj1u6RBTeEwhEwamc7PPr6AM63t3Pmj1ZRXNfpdkkiv7Dhaz1//aA3OwS/uXkhxTprfJQ16CochZkZhJk/es4iOIHzoR++w86jmICS2bTxcy4cfWkNqYgK/uvcqJo9K97ukIUHhMARNHZ3B059cRHJiAssfeoe1+0/5XZJIt/64t5qPPbyW7OHJPH3vVYzPG+53SUOGwmGImpA/gqc/eRV56Sl87JF1/GbTEb9LEvkLv1x7mI//dD3FOWk8/cmrKMrWUFI0KRyGsOKcNJ791GKuKMnic09t5juvlhE6s7qIf4JBx3+8uIv/+dw2rpmcxzOfWqzJZx8oHIa4rLRkfn73Qu6YW8i3Xt3Lpx/fSENzm99lyRBVe6aVjz+2nh+t2s9HF5Xw8F3zGZGiEzn4QeEgJCcm8F93zuZLt0zj5Z0nuO2/39bBchJ1WyrquPX/vcXq8lN87fYZfG3ZDBID+ojyi9a8AGBm/P21E3ji7xdxpqWd27/3Nr9Yc0jDTBJxwaDj4T/t584fvgPAr+49NJXsAAAMlUlEQVS9io8tGqtTYvhM4SB/YcH4HH73j1dz5bgcvvyb7fztT9Zz4nSz32XJIFVZ28TfPLyGf/v9Lq69LI/f/cPVzC7O8rssASxe/zKcP3++Ky0t9buMQSsYdPxi7SH+94u7SEkM8OX3TeOD84r015wMiI6g44l1h/n6S7sB+Nf3T+dOvb8izsw2OOfm96qvwkEuZH91I198ZisbDtVy5bhs/u32mUwZrYOQpO+2H6nnS7/ZzpaKOhZPzOUbH5ilI56jROEgAyoYdPxqQwX/8dJuGpvb+eiisfzDdZPIHZHid2kSR6pON/OtV8t4av1hcoan8OX3TWPZnDHaWogihYNERM2ZVr65cg9Pl1YwLCnAJ6+dwN3XjCctWbsaSs8amtt4aNV+Hv7TAdqDQT6ycCyfv+EyMofpWs/RpnCQiCqvauT//GE3L+88QXZaEh9/13juumocmWn6zy5/VnOmlZ+8fYCfrj5IQ3M7t84q4Is3TmFsrk6B4ReFg0TFxsO1fP+Ncl7dVcWIlEQ+dGUxH1s0lnE6/82QVl7VwM/fOcTTpZU0t3dw0+Wjue+9k5hRmOl3aUOewkGiatex0/zwj/v4/dZjtAcd116Wz98sKOG9U/NJSdRVuoaC5rYOXt11gsfXHOad/adIDiRw25wx3PvuCUwaqR0YYoXCQXxRdbqZJ9ZV8Mt1hzhxuoXMYUncOquAZXMKmTc2m0CCJh4Hk/aOIOsO1PD85qO8uO0YDS3tFGYN4yOLSvjQ/GLtsBCDFA7iq/aOIH8qP8lzG4/w8s7jNLcFyR2ezPXTRnH99FEsmpBDeqrmJ+JR/dk23tl3kld2VvHa7hPUNbWRlhzgphmjueOKIq6amKs/AmLYpYSDdjORAZcYSOC9U0by3ikjaWxp543dVby88wS/33aMp0orSEwwrijJYvHEPK4cl8OckiydXC1GnW5uY9PhOkoP1vBW+Um2VNQRdJCRmsiSaaNYOn0U756Srz3WBqGY2XIws5uA7wAB4GHn3Ncv1F9bDvGnpb2DDYdqeavsJG+Vn2TbkXqcgwSDKaMzmFmYwYzCTKYXZDB5VLp2dYyyuqZW9p5oZMfRenYcPc32I/XsOdFw7nc0qyiLaybncfWkPOaOzSZJJ8WLO3E3rGRmAWAvcANQCawHPuyc29nTYxQO8a+h86/SQ7VsOlzLjqOnqTnTem75yPQUJo0cwdjc4ZTkpDE2N42CzFQKMoeRn56i4YtL1N4RpLqxhWP1zRytO8vhmiYOn2ri4KkzlFed4WRjy7m+eSNSmFGYwRXF2cwfl82c4iyGa+su7sXjsNICoNw5tx/AzJ4ElgE9hoPEv/TUJK69LJ9rL8sHwDnH8dPN7Dx6mvKqRsqqGtlX3cjLO45zKiw0AAIJRu7wZPJGpJCXnkJ2WhLZaclkDksiPTWRjNTQ97SURIYnBxiWHGBYUoCUpACpiQkkJyaQFEggOZBAQoyHTDDoaO0I0tYRpKU99NXc1sHZ1g7OtnXQ1NrBmZZ2GprbaGhu53RzO/VNrdQ2tVHb1MrJxlaqG1qoOdNC8Ly/BfNGJFOSk8Z1U/OZPDKdSSNHMH1Mhi6uIzETDoVARdj9SmChT7WIT8yMgsxhFGQOY8m0UX+xrKG5jYqasxyrP8ux+maO1zdT3dDCycbQ18GTZ6hrauV0c/slv24gwUhMMJICCQQSjECCkWBGIAESLHQ7IQEMI8FCdRqAlynm1R7WROdnsHPu3G1cqN05R9CBwxEMQtA5gs7R4d1u7wjSEXS0e18d53+i90LmsCSyvMAszBrG7KJM8tNTGJ2ZypjMYYzOTKUkJ01bA9KjWHlndPenW5f/EWZ2D3APQElJSaRrkhiSnprE9DFJTB+TccF+7R1BzrR0cNr7K/psWztnWkJ/Xbe0d9Dc1kFzW5DW9iCtHaHv7cEg7R3eh3FHkI7OD+qg8z64Qx/anR/0Qce561y4c/+EPuzDWVh6dL7BzbyAOXc7dD8UQKGQ6gyoxMCfQyspkECKt8WTmhggJSmB1KQAacmdX4mkpyaSnpLEiNREDblJv8VKOFQCxWH3i4Cj53dyzj0EPAShOYfolCbxJDGQQGZagk7lIdJPsbK7wXpgspmNN7NkYDnwgs81iYgMWTGx5eCcazezzwArCe3K+qhzbofPZYmIDFkxEQ4AzrkXgRf9rkNERGJnWElERGKIwkFERLpQOIiISBcKBxER6ULhICIiXcTEiff6wsyqgUN9fHgecHIAyxkoquvSqK5Lo7ouzWCsa6xzLr83HeM2HPrDzEp7e2bCaFJdl0Z1XRrVdWmGel0aVhIRkS4UDiIi0sVQDYeH/C6gB6rr0qiuS6O6Ls2QrmtIzjmIiMiFDdUtBxERuYBBGw5mdqeZ7TCzoJnNP2/ZA2ZWbmZ7zOzGHh4/3szWmlmZmT3lnUp8oGt8ysw2e18HzWxzD/0Omtk2r1/EL5xtZl8xsyNhtd3SQ7+bvHVYbmb3R6Gub5rZbjPbambPmVlWD/2isr4u9vObWYr3Oy733kvjIlVL2GsWm9kbZrbLe/9/tps+7zGz+rDf779Gui7vdS/4e7GQ73rra6uZzY1CTVPC1sNmMzttZp87r09U1peZPWpmVWa2Pawtx8xe8T6HXjGz7B4eu8LrU2ZmKwakINd5hatB9gVMA6YAbwLzw9qnA1uAFGA8sA8IdPP4p4Hl3u0fAp+KcL3/BfxrD8sOAnlRXHdfAf7HRfoEvHU3AUj21un0CNe1FEj0bn8D+IZf66s3Pz/waeCH3u3lwFNR+N0VAHO92+nA3m7qeg/wu2i9n3r7ewFuAV4idKG8RcDaKNcXAI4TOhYg6usLuBaYC2wPa/s/wP3e7fu7e88DOcB+73u2dzu7v/UM2i0H59wu59yebhYtA550zrU45w4A5cCC8A4WuiDwdcAzXtNjwO2RqtV7vb8GnojUa0TAAqDcObffOdcKPElo3UaMc+5l51znRaLXELpioF968/MvI/TegdB7aYl1Xmw6Qpxzx5xzG73bDcAuQtdojwfLgJ+5kDVAlpkVRPH1lwD7nHN9Pbi2X5xzq4Ca85rD30M9fQ7dCLzinKtxztUCrwA39beeQRsOF1AIVITdr6Trf55coC7sg6i7PgPpGuCEc66sh+UOeNnMNnjX0Y6Gz3ib9o/2sCnbm/UYSR8n9Fdmd6Kxvnrz85/r472X6gm9t6LCG8a6AljbzeKrzGyLmb1kZpdHqaSL/V78fk8tp+c/0PxYXwCjnHPHIBT8wMhu+kRkvcXMxX76wsxeBUZ3s+hLzrnne3pYN23n77LVmz690ssaP8yFtxre5Zw7amYjgVfMbLf3V0afXagu4AfA1wj9zF8jNOT18fOfopvH9nvXt96sLzP7EtAOPN7D0wz4+uqu1G7aIvY+ulRmNgL4NfA559zp8xZvJDR00ujNJ/0GmByFsi72e/FzfSUDtwEPdLPYr/XVWxFZb3EdDs656/vwsEqgOOx+EXD0vD4nCW3SJnp/8XXXZ0BqNLNE4A5g3gWe46j3vcrMniM0pNGvD7verjsz+zHwu24W9WY9Dnhd3mTbrcAS5w24dvMcA76+utGbn7+zT6X3e86k67DBgDOzJELB8Lhz7tnzl4eHhXPuRTP7vpnlOecieh6hXvxeIvKe6qWbgY3OuRPnL/BrfXlOmFmBc+6YN8RW1U2fSkLzIp2KCM219stQHFZ6AVju7UkyntBfAOvCO3gfOm8AH/SaVgA9bYn01/XAbudcZXcLzWy4maV33iY0Kbu9u74D5bxx3r/q4fXWA5MttFdXMqFN8hciXNdNwL8AtznnmnroE6311Zuf/wVC7x0IvZde7ynQBoo3p/EIsMs593976DO6c+7DzBYQ+hw4FeG6evN7eQG4y9traRFQ3zmkEgU9br37sb7ChL+HevocWgksNbNsbwh4qdfWP5Gegffri9CHWiXQApwAVoYt+xKhPU32ADeHtb8IjPFuTyAUGuXAr4CUCNX5U+De89rGAC+G1bHF+9pBaHgl0uvu58A2YKv35iw4vy7v/i2E9obZF6W6ygmNrW72vn54fl3RXF/d/fzAVwmFF0Cq994p995LE6Kwjq4mNKSwNWw93QLc2/k+Az7jrZsthCb2F0ehrm5/L+fVZcD3vPW5jbC9DCNcWxqhD/vMsLaory9C4XQMaPM+u+4mNEf1GlDmfc/x+s4HHg577Me991k58HcDUY+OkBYRkS6G4rCSiIhchMJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLv4/Uw5YHte+pP0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# cem test 2\n",
    "# cem fits the x of f(x), in this approach, the x can be parameters of \n",
    "# the function f, therefore f can be dependent on other variables z like f(x,z). It works \n",
    "# as long as there are minimum of the function f.\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "N=100\n",
    "ratio=0.1\n",
    "num_itr=50\n",
    "\n",
    "f=lambda x: 3*x**2+x**4  # function of f(x)\n",
    "samples = np.random.normal(0,1,N) # x-value of f(x)\n",
    "print('true min: ', np.min(f(samples)))\n",
    "for i in range(num_itr):\n",
    "    # sample and select top 10% (max [-a:]or min [:a])\n",
    "    selected_samples = samples[np.array(f(samples)).argsort()[:int(N*ratio)]]\n",
    "    m=np.mean(selected_samples)\n",
    "    std=np.std(selected_samples)  # std decrease fastly\n",
    "#     print(std)\n",
    "    # fit the gaussian\n",
    "    samples=np.random.normal(m, std, N)\n",
    "print(f(selected_samples))\n",
    "\n",
    "'''display the function f(x) '''\n",
    "x=np.linspace(-10,10,N)\n",
    "y=f(x)\n",
    "plt.plot(x,y)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional results from above:\n",
    "* CE method can also fit on the parameters of a function, like x of f(x), which actually indicates the fitting can be done on any parameterized variables of the original variable (as criteria value) for CE, and on any re-parameterized variables for the previous parameterized variables, ie:\n",
    "\n",
    "    f->f(x)->f(g(z))->...\n",
    "    \n",
    "    where the Gaussian distribution can be fitted on f or x or z, ... respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes for applying CE method in practice:\n",
    "* Although from above analysis, CE method can fit on any parameterized variables of the criteria function, there is actually a preference of CE method for which variables to fit on:\n",
    "\n",
    "    First, the main idea of CE method is **to transform the original optimization problem**, which is on the criteria function, **to an associated stochastic problem in another space**, which is the parameterized variable space, **for applying the sampling-based iterative method to get the optimum in original criteria function**.\n",
    "    \n",
    "    And the reason of applying this transformation is that **if directly applying the sampling-based interative evaluation on original criteria value function (like in CE method test 1), the probability of drawing a higg-valued smaple is presumably very low**, therefore finding the optimal samples from naive sampling is intractable. This is for the case that the criteria function is highly complicated and non-convex.\n",
    "    \n",
    "    Therefore, the sampling space transformation is to make the probability of drawing high-valued (criteria value) samples larger than direcly drawing from the criteria function. How to guarantee that, w.r.t. the transformation function?\n",
    "    \n",
    "    **So a linear transformation will probabily not help with changing the map of distribution/function values, therefore not help with making the probability of sampling large criteria values,** like g(x)=w\\*x+b.\n",
    "    \n",
    "    CE method also has disadvantages in **sensitivity to initialization of the Gaussian distribution**. A non-proper initialization could make CE method converge to local optimum easily. For example, in algorithm QT-Opt, if CE method isn't initialized properly, the several optimum reached by different runnings of CE method for the same case could make the Q-network applying CE not converge.\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Noisy CE method\n",
    "Simply adding an off-set on the standard deviation term in distribution of CE method to prevent it from decreasing too fast. It is basically a boosting of exploration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('true min: ', 0.0003412901149159018)\n",
      "[0.00034129 0.00346261 0.0039516  0.00562357 0.01831646 0.02385421\n",
      " 0.02464838 0.02520516 0.03229293 0.03421347]\n",
      "[0.0001847  0.00044797 0.0006746  0.00076897 0.00079741 0.00105112\n",
      " 0.00106451 0.00110887 0.00120241 0.00142378]\n",
      "[1.03419228e-06 2.38243375e-06 3.90924894e-05 7.79811514e-05\n",
      " 8.99536363e-05 1.59339597e-04 1.61820122e-04 1.71939323e-04\n",
      " 2.71328281e-04 2.83190690e-04]\n",
      "[1.39912707e-06 9.08494517e-06 2.40212221e-05 3.15651967e-05\n",
      " 1.42864904e-04 3.03949594e-04 4.55235702e-04 4.92096930e-04\n",
      " 5.96491898e-04 6.13972094e-04]\n",
      "[3.35399938e-06 1.21945547e-05 1.25835656e-05 2.18142469e-05\n",
      " 2.99596319e-05 9.00015199e-05 1.21487414e-04 1.87495442e-04\n",
      " 2.11307611e-04 2.60249908e-04]\n",
      "[1.12668277e-06 1.79999886e-05 3.97441648e-05 6.02497161e-05\n",
      " 9.23291935e-05 9.79880099e-05 9.92044171e-05 1.81352091e-04\n",
      " 2.42245264e-04 2.73295046e-04]\n",
      "[6.50466348e-07 4.40883648e-05 6.78813587e-05 1.04186431e-04\n",
      " 1.50469673e-04 1.79026235e-04 1.93043327e-04 2.08465794e-04\n",
      " 5.06903521e-04 8.75083687e-04]\n",
      "[8.32305501e-10 1.39793015e-06 2.19462905e-05 2.79789584e-05\n",
      " 3.62515543e-05 5.34408719e-05 1.51061842e-04 1.67260825e-04\n",
      " 2.12328199e-04 3.00972854e-04]\n",
      "[3.24458909e-06 3.57956532e-06 7.86982126e-06 1.07796230e-05\n",
      " 2.19359405e-05 7.94978531e-05 1.57252918e-04 1.67963866e-04\n",
      " 2.14647448e-04 3.11392496e-04]\n",
      "[8.64130884e-07 4.95059157e-06 6.70243834e-06 1.77568946e-05\n",
      " 3.77815400e-05 6.29971248e-05 7.96946320e-05 8.23540853e-05\n",
      " 1.00669053e-04 1.19927870e-04]\n",
      "[2.89399617e-05 3.06896580e-05 4.66619090e-05 5.94656983e-05\n",
      " 7.24398934e-05 8.46828396e-05 1.66964310e-04 1.93298739e-04\n",
      " 2.10396788e-04 2.10621271e-04]\n",
      "[1.94153752e-06 1.18948630e-04 1.41905044e-04 1.72528123e-04\n",
      " 2.25464763e-04 2.42414064e-04 2.47946494e-04 3.11273161e-04\n",
      " 4.34866775e-04 4.73459456e-04]\n",
      "[2.05705541e-07 3.24318565e-06 1.11801580e-05 3.97272835e-05\n",
      " 7.31821707e-05 1.08158910e-04 1.08272903e-04 1.14287149e-04\n",
      " 1.30834848e-04 2.17202510e-04]\n",
      "[2.29647179e-08 7.99365357e-07 3.61848705e-05 6.10447105e-05\n",
      " 8.51583760e-05 1.41449336e-04 1.71768457e-04 1.77245799e-04\n",
      " 1.93315939e-04 2.65782795e-04]\n",
      "[7.15287146e-08 8.08571871e-08 2.92248137e-05 5.79443735e-05\n",
      " 1.13317482e-04 1.16182186e-04 1.71255018e-04 3.95513090e-04\n",
      " 4.41437209e-04 5.81416750e-04]\n",
      "[4.20067046e-06 7.77142134e-06 2.19799388e-05 4.68359839e-05\n",
      " 8.99047233e-05 4.83527524e-04 7.46750582e-04 8.10750610e-04\n",
      " 1.17557163e-03 1.65018856e-03]\n",
      "[6.60715074e-08 3.45901717e-06 8.11565975e-06 1.01059250e-05\n",
      " 1.41881342e-05 2.28123344e-05 5.71845247e-05 9.19735708e-05\n",
      " 9.34531466e-05 1.90908386e-04]\n",
      "[7.58694488e-08 2.25156263e-07 2.14642930e-05 9.30636336e-05\n",
      " 1.16334463e-04 1.99247926e-04 2.98393789e-04 6.40971495e-04\n",
      " 9.91115556e-04 1.09059540e-03]\n",
      "[1.86180988e-06 1.87347814e-05 2.57739870e-05 2.81291589e-05\n",
      " 3.31311070e-05 8.22813657e-05 1.00391618e-04 1.32344186e-04\n",
      " 1.88982300e-04 2.14196266e-04]\n",
      "[1.48620330e-05 1.79305487e-05 1.45489946e-04 2.61029557e-04\n",
      " 3.12607352e-04 3.14615254e-04 4.98419341e-04 5.51778287e-04\n",
      " 7.02761295e-04 8.06422174e-04]\n",
      "[7.26386503e-06 1.34713762e-04 1.80144850e-04 2.10024000e-04\n",
      " 2.50021312e-04 3.36753740e-04 3.85093924e-04 4.49323770e-04\n",
      " 6.28705082e-04 6.66907235e-04]\n",
      "[7.87038422e-07 2.24881619e-05 2.82306935e-05 4.54162632e-05\n",
      " 6.36889744e-05 7.95572245e-05 4.94637005e-04 5.47508727e-04\n",
      " 7.24960546e-04 7.50222129e-04]\n",
      "[1.29836722e-06 2.44719907e-04 2.46139672e-04 3.53104734e-04\n",
      " 4.12255794e-04 4.92354669e-04 6.19354998e-04 7.04673262e-04\n",
      " 9.22657595e-04 9.69114945e-04]\n",
      "[3.15254867e-06 1.18922765e-05 9.89862896e-05 1.24100179e-04\n",
      " 1.32900612e-04 1.46892756e-04 1.54113831e-04 1.89713441e-04\n",
      " 2.56058817e-04 3.29342724e-04]\n",
      "[1.33946756e-06 2.24418178e-06 2.96205702e-06 3.86279081e-06\n",
      " 3.71078580e-05 4.51280370e-05 1.46643497e-04 1.74297928e-04\n",
      " 2.61467870e-04 3.33421638e-04]\n",
      "[2.32222123e-07 3.66716914e-06 8.17202418e-06 2.68501628e-05\n",
      " 4.48236102e-05 5.97089282e-05 9.97296677e-05 1.23318486e-04\n",
      " 2.18439928e-04 2.33003729e-04]\n",
      "[9.08523611e-07 2.84834916e-06 5.93088018e-06 2.99458740e-05\n",
      " 8.93982390e-05 2.05852325e-04 2.99654831e-04 3.21386356e-04\n",
      " 3.95570798e-04 4.43489120e-04]\n",
      "[7.82086890e-08 1.47627251e-05 1.04882038e-04 1.66380714e-04\n",
      " 2.05238759e-04 5.07035201e-04 5.25825108e-04 8.73629762e-04\n",
      " 9.75735478e-04 1.12958896e-03]\n",
      "[5.55191274e-05 1.46890908e-04 1.58393112e-04 3.03293424e-04\n",
      " 3.26337107e-04 4.00293602e-04 4.31871749e-04 5.17227848e-04\n",
      " 5.72505364e-04 7.33902134e-04]\n",
      "[4.28990537e-06 6.98523030e-06 1.15208272e-05 1.68375804e-05\n",
      " 2.78182235e-05 3.31076807e-05 5.08711289e-05 1.08789850e-04\n",
      " 1.48370461e-04 1.75516914e-04]\n",
      "[3.33435644e-06 3.71208832e-06 3.50962462e-05 1.17859363e-04\n",
      " 1.41468344e-04 1.48411049e-04 1.83321383e-04 3.49029252e-04\n",
      " 4.02289730e-04 4.15877531e-04]\n",
      "[1.40438120e-06 7.33888712e-06 7.16367144e-05 1.04091655e-04\n",
      " 1.29241745e-04 1.70824851e-04 2.41922282e-04 2.47291509e-04\n",
      " 2.88418717e-04 5.20775402e-04]\n",
      "[6.89417233e-07 4.53997645e-06 4.85286628e-06 1.30692555e-04\n",
      " 1.32657939e-04 1.46913665e-04 2.14330478e-04 2.43167865e-04\n",
      " 2.71629504e-04 6.01274131e-04]\n",
      "[1.04211738e-05 2.27426853e-05 2.62139547e-05 4.91908469e-05\n",
      " 5.94656874e-05 1.58601142e-04 3.16851012e-04 3.63572539e-04\n",
      " 3.69409782e-04 6.15400365e-04]\n",
      "[5.22092995e-06 5.34220017e-06 2.36458988e-05 2.62374762e-05\n",
      " 2.93855329e-05 4.36948972e-05 5.84324160e-05 6.14323310e-05\n",
      " 1.39115626e-04 7.51196086e-04]\n",
      "[1.67546504e-06 1.50407999e-05 6.51015864e-05 1.55228979e-04\n",
      " 1.73165932e-04 1.80444648e-04 1.88234726e-04 2.27789141e-04\n",
      " 2.95114428e-04 5.45292484e-04]\n",
      "[2.66992742e-05 3.27941377e-05 1.07449476e-04 1.37686341e-04\n",
      " 1.39821083e-04 2.56283448e-04 3.52767744e-04 3.70224606e-04\n",
      " 3.89859108e-04 4.03637316e-04]\n",
      "[2.32343436e-08 1.21337654e-07 9.51640088e-06 5.14394948e-05\n",
      " 1.19633688e-04 3.23882969e-04 5.01850020e-04 5.02253257e-04\n",
      " 5.89464762e-04 6.46291794e-04]\n",
      "[1.66635023e-05 1.79080811e-05 3.23236193e-05 3.79594380e-04\n",
      " 3.82441337e-04 4.21250770e-04 4.32168789e-04 4.57780365e-04\n",
      " 6.44221822e-04 6.78974868e-04]\n",
      "[4.42417541e-06 7.49070027e-06 3.58121188e-05 4.87212040e-05\n",
      " 6.06331339e-05 8.69283873e-05 1.08239833e-04 2.08747985e-04\n",
      " 3.27842675e-04 5.56779383e-04]\n",
      "[1.05693010e-05 1.00449930e-04 1.47753907e-04 1.67087746e-04\n",
      " 1.89427591e-04 2.83537556e-04 4.41028377e-04 5.62772913e-04\n",
      " 6.01684211e-04 6.25175551e-04]\n",
      "[3.45828350e-06 2.65757320e-05 3.78370746e-05 1.89959390e-04\n",
      " 2.31513414e-04 2.56922535e-04 3.12283035e-04 3.17569180e-04\n",
      " 3.44752152e-04 3.79594911e-04]\n",
      "[1.30998887e-05 3.37540254e-05 5.99401848e-05 7.77140516e-05\n",
      " 1.46775166e-04 1.52340029e-04 2.36324144e-04 3.17231512e-04\n",
      " 3.50982889e-04 3.81689567e-04]\n",
      "[1.09602361e-05 1.24245213e-05 2.39006906e-05 3.49310946e-05\n",
      " 5.33566468e-05 1.59749169e-04 2.68439464e-04 3.97239336e-04\n",
      " 4.00216034e-04 4.15501854e-04]\n",
      "[1.74133568e-05 1.87260585e-05 3.61588167e-05 5.19479391e-05\n",
      " 7.76485351e-05 1.08216480e-04 1.15235470e-04 1.95119615e-04\n",
      " 2.81095804e-04 3.32865893e-04]\n",
      "[1.29187022e-05 1.34081809e-05 1.60310926e-05 2.87351447e-05\n",
      " 4.22376612e-05 4.72156757e-05 1.08785980e-04 2.90086482e-04\n",
      " 3.00239648e-04 3.08375629e-04]\n",
      "[8.67072283e-06 9.84622607e-06 3.25506164e-05 3.14911888e-04\n",
      " 3.56941340e-04 3.74981504e-04 5.54842440e-04 1.20767409e-03\n",
      " 1.34431622e-03 1.67320593e-03]\n",
      "[2.78825960e-06 2.44064591e-05 4.94115317e-04 7.44056964e-04\n",
      " 1.09426154e-03 1.09949219e-03 1.25816702e-03 1.26162612e-03\n",
      " 1.35163524e-03 1.45062873e-03]\n",
      "[1.46373716e-05 2.59501633e-05 3.52707588e-05 2.10716489e-04\n",
      " 2.60392928e-04 2.87804101e-04 3.14974915e-04 8.03546933e-04\n",
      " 8.52132762e-04 1.00027697e-03]\n",
      "[1.11919516e-06 1.27960083e-06 1.04978534e-05 2.08510115e-04\n",
      " 2.74293276e-04 3.46596885e-04 3.52836470e-04 4.09263340e-04\n",
      " 6.13713082e-04 6.50898399e-04]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8XGed7/HPT6Nm2eqSbVnFPS5xi+3YjkkCxIlTCHE2kMUsEO8SNgTCLmUvu8mF14YLu3fhsnspd2khCQQIKYSEBEhwOiZxXOTeLblKbpKtYsmy6jz3jzkygyXZsqSZMyN936+XrJnnPDPz09F4vjrPc4o55xAREQmX4HcBIiISexQOIiLShcJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLhL9LqCv8vLy3Lhx4/wuQ0QkbmzYsOGkcy6/N33jNhzGjRtHaWmp32WIiMQNMzvU274aVhIRkS4UDiIi0oXCQUREulA4iIhIFwoHERHpQuEgIiJdKBxERKSLIRUOre1BfvDmPlbtrfa7FBGRS/bqzhM8tGof0bi885AKh6SA8dCqffx+6zG/SxERuWTPbTrCz9ccwswi/lpDKhzMjBmFmWw7Uu93KSIil2zbkXpmFmZG5bUuGg5m9qiZVZnZ9rC2HDN7xczKvO/ZXruZ2XfNrNzMtprZ3LDHrPD6l5nZirD2eWa2zXvMdy3CkTirKJO9JxpobuuI5MuIiAyouqZWDtc0MbMwKyqv15sth58CN53Xdj/wmnNuMvCadx/gZmCy93UP8AMIhQnwILAQWAA82BkoXp97wh53/msNqJmFmbQHHbuPN0TyZUREBtT2I6cBYmfLwTm3Cqg5r3kZ8Jh3+zHg9rD2n7mQNUCWmRUANwKvOOdqnHO1wCvATd6yDOfcOy40w/KzsOeKiJlFodTdVlkXyZcRERlQW4+EPrNiJhx6MMo5dwzA+z7Say8EKsL6VXptF2qv7KY9YsZkppIzPFnzDiISV7YfqackJ43MtKSovN5AT0h3N1/g+tDe/ZOb3WNmpWZWWl3dt91RzYyZhZlsrVQ4iEj82FpZz8yi6Gw1QN/D4YQ3JIT3vcprrwSKw/oVAUcv0l7UTXu3nHMPOefmO+fm5+f36noV3ZpZmElZVaMmpUUkLtSeaaWy9mzUhpSg7+HwAtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0UuBld6yBjNb5O2ldFfYc0XMzKJMOoKOncdOR/qlRET6rXMYfFYshYOZPQG8A0wxs0ozuxv4OnCDmZUBN3j3AV4E9gPlwI+BTwM452qArwHrva+vem0AnwIe9h6zD3hpYH60nnWm73bNO4hIHOgMh8ujGA4XvUyoc+7DPSxa0k1fB9zXw/M8CjzaTXspMONidQykgsxU8kYka95BROLCtsp6xuWmkTksOpPRMMSOkO7UeaS0thxEJB5sO1LPjChuNcAQDQcIjd3tPdHA2VZNSotI7DrV2MKRurPMiuKeSjCEw2FGYSZBBzuPaetBRGJX53yDthyiZNa5I6UVDiISu7YrHKJrVEYK+ekpmpQWkZi2tbKe8XnDyUiN3mQ0DOFwMDNmF2WyWedYEpEY5Zxjc0Uds6M83wBDOBwA5hRnsb/6DPVNbX6XIiLSxbH6ZqoaWphTHJ3TdIcb4uEQOmv4Fm09iEgM2lIR+myaU5J9kZ4Db0iHw6ziTMz+/AsQEYklmyvqSA4kMK0gPeqvPaTDISM1iYn5I9iscBCRGLSpoo5pYzJISQxE/bWHdDhAaN5hc0UdoTN/iIjEhvaOINsq67nCh/kGUDgwuziLU97pcEVEYkVZVSNn2zp8mYwGhcO5VN6koSURiSGdw90KB59MGZ1OSmKCJqVFJKZsPlxHVloSY3PTfHn9IR8OSYEEZhZmalJaRGJK6OC3LELXQYu+IR8OENps236knraOoN+liIjQ2NLO3qoG34aUQOEAhCalW9qD7D7W4HcpIiJsq6zHOZhTonDwVWc6b66o9bkSEZE/T0bPLlI4+Kooexh5I5K1x5KIxITNFbWMzU0jZ3iybzUoHAidofWKkmw2HVY4iIi/nHNsPFzn28FvnRQOnnljszlw8gynGlv8LkVEhrDK2rNUN7Qwb1yOr3UoHDzzxobOerhRWw8i4qPSQzUAzPPhTKzhFA6emYWZJAXs3C9GRMQPGw7VMiIlkSmjo38m1nAKB09qUoAZhZlsPKQ9lkTEPxsO1XFFSRaBBH8OfuukcAgzrySbLZX1tLbrYDgRib6G5jb2HD/NXJ+HlEDh8Bfmjc2mtT3IjqP1fpciIkPQ5oo6gg7mj1M4xJTOSekNGloSER9sOFSLmX9nYg2ncAgzMiOV4pxhCgcR8cWGQ7VMGZVOemqS36UoHM43rySb0kO1ujKciERVR9Cx6XBdTAwpgcKhi3ljs6luaNGV4UQkqvaeaKCxpf3c8Lbf+hUOZvZ5M9thZtvN7AkzSzWz8Wa21szKzOwpM0v2+qZ498u95ePCnucBr32Pmd3Yvx+pf+aNDR2VqKElEYmmUu8zZ16Jv0dGd+pzOJhZIfCPwHzn3AwgACwHvgF8yzk3GagF7vYecjdQ65ybBHzL64eZTfcedzlwE/B9Mwv0ta7+mjI6neHJAYWDiETVxkO15KenUJwzzO9SgP4PKyUCw8wsEUgDjgHXAc94yx8DbvduL/Pu4y1fYqFLHC0DnnTOtTjnDgDlwIJ+1tVngYTQSfjWH9SR0iISPesP1jCvJNu3K7+dr8/h4Jw7AvwncJhQKNQDG4A651y7160SKPRuFwIV3mPbvf654e3dPOYvmNk9ZlZqZqXV1dV9Lf2iFozPYc+JBuqaWiP2GiIinY7UnaWy9iwLxsfGkBL0b1gpm9Bf/eOBMcBw4OZuunbu9tNdHLoLtHdtdO4h59x859z8/Pz8Sy+6lxaMz8E5KD2ooSURibz1B0IjFYMiHIDrgQPOuWrnXBvwLLAYyPKGmQCKgKPe7UqgGMBbngnUhLd38xhfzCnOIjmQwDoNLYlIFKw9UEN6aiLTCjL8LuWc/oTDYWCRmaV5cwdLgJ3AG8AHvT4rgOe92y949/GWv+5CBxO8ACz39mYaD0wG1vWjrn5LTQowpziLtQcUDiISeesOnOLKcTm+n2wvXH/mHNYSmljeCGzznush4F+AL5hZOaE5hUe8hzwC5HrtXwDu955nB/A0oWD5A3Cfc66jr3UNlAXjc9h+pJ4zLe0X7ywi0kcnG1vYV30mpoaUILS3UZ855x4EHjyveT/d7G3knGsG7uzhef4d+Pf+1DLQFozP4b/fKGfj4VqumRy5+Q0RGdpicb4BdIR0j+aOzSaQYKzT0JKIRNDaAzUMSwowY0ym36X8BYVDD0akJDJjTAZr9yscRCRy1h6oYe7YLJITY+vjOLaqiTELJ+SyuaKO5jbfp0BEZBCqb2pj9/HTLByf63cpXSgcLmDBuBxaO4JsqajzuxQRGYRKD9XgXOzNN4DC4YKuHJeDGZp3EJGIWHeghuRAQkxc3Od8CocLyExLYsqodB3vICIRsfZADbOLM0lN8u1coz1SOFzEogm5lB6qobU96HcpIjKINDS3se1IfUzON4DC4aIWT8yluS3IZs07iMgAWn+who6gY/EkhUNcWjghlwSDt8tP+l2KiAwib5efIjkxgbklsXHlt/MpHC4ic1gSMwozeWffKb9LEZFBZPW+U8wfmx2T8w2gcOiVxRPz2FRRS1OrzrMkIv1Xc6aVXcdO865JeX6X0iOFQy8snphLW4djva7vICIDoHMk4qqJsTnfAAqHXpk/LpukgLF6n+YdRKT/Vu87yYiURGYVxtb5lMIpHHohLTmRK0qyWV2ueQcR6b/V+06xcHwOiYHY/QiO3cpizOKJuWw/Wk99U5vfpYhIHDtad5YDJ8/E9JASKBx6bfHEPJyDNQe09SAifdc537B4YuxORoPCodfmFGcxLCnAah3vICL98Pa+k+QMT2bq6HS/S7kghUMvJScmcOX4HN7W8Q4i0kfOOd7Zd4qrJuSSEEPXi+6OwuESXD0pl/KqRo7Vn/W7FBGJQ/uqGzlW3xzTxzd0UjhcgmsvC11L+k97NbQkIpfuj95nx7WXKRwGlSmj0hmVkcIfy6r9LkVE4tCqvdVMyB9OUXaa36VclMLhEpgZ10zO562yk3QEnd/liEgcaW7rYO2BU1w7Od/vUnpF4XCJrr0sn/qzbWyt1Cm8RaT31h+sobktyLsvUzgMSldPysMMVmneQUQuwaq91SQHElg4IfauF90dhcMlyhmezMzCTFZp3kFELsGqvSeZPy6btOREv0vpFYVDH1w7OZ/NFXXUn9WpNETk4o7XN7PnRMO5PR7jgcKhD669LJ+OoNPR0iLSK50jDfEyGQ0Khz65oiSLESmJrCpTOIjIxf2p7CT56SlMK4jtU2aEUzj0QVIggcUTc1m1txrntEuriPSsI+h4q6yaaybnYRbbp8wI169wMLMsM3vGzHab2S4zu8rMcszsFTMr875ne33NzL5rZuVmttXM5oY9zwqvf5mZrejvDxUN75kykiN1ZymvavS7FBGJYVsq66htaoubXVg79XfL4TvAH5xzU4HZwC7gfuA159xk4DXvPsDNwGTv6x7gBwBmlgM8CCwEFgAPdgZKLLtu6kgAXttd5XMlIhLLXt9VRSDBeM9lI/0u5ZL0ORzMLAO4FngEwDnX6pyrA5YBj3ndHgNu924vA37mQtYAWWZWANwIvOKcq3HO1QKvADf1ta5oGZ2ZyuVjMnh9l8JBRHr22u4q5o/NJjMtye9SLkl/thwmANXAT8xsk5k9bGbDgVHOuWMA3vfOuCwEKsIeX+m19dQe866bOpLSQzXUNbX6XYqIxKCjdWfZdez0uZGGeNKfcEgE5gI/cM5dAZzhz0NI3eluJsZdoL3rE5jdY2alZlZaXe3/QWjXTR1J0MEf9/pfi4jEnte9Yecl04ZWOFQClc65td79ZwiFxQlvuAjve1VY/+KwxxcBRy/Q3oVz7iHn3Hzn3Pz8fP8nd2YXZZE7PJnXNLQkIt14fXcVJTlpTMwf4Xcpl6zP4eCcOw5UmNkUr2kJsBN4Aejc42gF8Lx3+wXgLm+vpUVAvTfstBJYambZ3kT0Uq8t5iUkGO+dOpI391TR3hH0uxwRiSFnWzt4u/wk100dGVe7sHbq70k+/gF43MySgf3A3xEKnKfN7G7gMHCn1/dF4BagHGjy+uKcqzGzrwHrvX5fdc7V9LOuqFkydSTPbKhkw6FaFk7I9bscEYkRq/edpKU9GJdDStDPcHDObQbmd7NoSTd9HXBfD8/zKPBof2rxy9WT80gKGK/vrlI4iMg5r+2uYnhygAXj4+MsrOfTEdL9lJ6axMLxuecmnkREnHO8sbuKaybnk5IY8LucPlE4DIDrpo6krKqRgyfP+F2KiMSAHUdPc6y+mevidEgJFA4D4obpowBYueO4z5WISCxYueM4CRaak4xXCocBUJyTxozCDIWDiAChcFgwPofcESl+l9JnCocBcuP00Ww8XMeJ081+lyIiPtpf3cjeE43cePlov0vpF4XDALlxRuiN8PLOEz5XIiJ+Wrkj9BmwVOEgAJNHjmBC3nBe1tCSyJC2csdxZhVlUpg1zO9S+kXhMEDMjKWXj+adfaeob9K1pUWGouP1zWyuqIv7ISVQOAyoGy8fRXvQ8dpuDS2JDEUv7wyNHNx4+SifK+k/hcMAml2UxeiMVO21JDJErdxxnIn5w5k0Mn6uFd0ThcMASkgwll4+ij/uraaptd3vckQkiuqaWlmzv2ZQDCmBwmHA3TRjNM1tQd7YrWs8iAwlK3ccpyPouHlGgd+lDAiFwwBbOD6X/PQUfrul20tSiMgg9dstxxibGzogdjBQOAywQIJxy4zRvLGnisYWDS2JDAUnG1tYve8kt84qiMtrN3RH4RAB7589hpb2IK/qgDiRIeGl7ccJutD//cFC4RABc0uyKchM1dCSyBDx2y1HmTRyBFNGxf9eSp0UDhGQkGC8b2YBq8qqdUCcyCB34nQz6w/WDKohJVA4RMz7Z4+hrcOxcqeOeRAZzH6/9RjOwa2zBs+QEigcImZWUSYlOWkaWhIZ5H679SjTCjKYNHKE36UMKIVDhJgZ75tVwOp9pzjV2OJ3OSISAZW1TWw6XMetswbHsQ3hFA4RdNvsMXQEHb/beszvUkQkAp7fHBoZeP8gG1IChUNETSvIYFpBBs9urPS7FBEZYM45fr2xkgXjcijJTfO7nAGncIiwD8wtZEtlPeVVDX6XIiIDaEtlPfurz3DH3EK/S4kIhUOE3TZnDAkGz2484ncpIjKAnt1YSXJiArcMwvkGUDhE3Mj0VK69LJ/nNh0hGHR+lyMiA6C1PcgLW46ydPooMlKT/C4nIhQOUXDH3CKO1TezZv8pv0sRkQHwxp4q6pra+MDcIr9LiRiFQxQsnT6K9JREfq2hJZFB4dmNleSNSOGayXl+lxIxCocoSE0K8L5ZBby0/ZguAiQS52rPtPL67ipunzOGxMDg/QgdvD9ZjLljbhFNrR28uE2n0xCJZ89vPkJbh+OvBuleSp36HQ5mFjCzTWb2O+/+eDNba2ZlZvaUmSV77Sne/XJv+biw53jAa99jZjf2t6ZYdOW4bCbkDefJdYf9LkVE+sg5x5PrK5hZmMnlYzL9LieiBmLL4bPArrD73wC+5ZybDNQCd3vtdwO1zrlJwLe8fpjZdGA5cDlwE/B9MwsMQF0xxcxYvqCY0kO1lJ3QMQ8i8WhzRR27jzfw4QUlfpcScf0KBzMrAt4HPOzdN+A64Bmvy2PA7d7tZd59vOVLvP7LgCedcy3OuQNAObCgP3XFqg/MLSIpYDyxrsLvUkSkD55Yd5i05AC3zRl8p8s4X3+3HL4N/DMQ9O7nAnXOuc5Z10qgc2CuEKgA8JbXe/3PtXfzmEEld0QKS6eP5tlNlTS3dfhdjohcgobmNn675RjvnzWGESmJfpcTcX0OBzO7Fahyzm0Ib+6mq7vIsgs95vzXvMfMSs2stLq6+pLqjRXLFxRT19TGyh2amBaJJy9sOcrZtg6WLyj2u5So6M+Ww7uA28zsIPAkoeGkbwNZZtYZq0VA5wUNKoFiAG95JlAT3t7NY/6Cc+4h59x859z8/Pz8fpTun3dNzKM4ZxhPamhJJK48ua6CqaPTmVOc5XcpUdHncHDOPeCcK3LOjSM0ofy6c+4jwBvAB71uK4DnvdsvePfxlr/unHNe+3Jvb6bxwGRgXV/rinUJCcbyK0t4Z/8pDpw843c5ItIL24/Us+1IPR9eUDKoLgV6IZE4zuFfgC+YWTmhOYVHvPZHgFyv/QvA/QDOuR3A08BO4A/Afc65QT0gf+e8IhITjMfXHPK7FBHphV+sOURqUgK3zxmU06HdGpBZFefcm8Cb3u39dLO3kXOuGbizh8f/O/DvA1FLPBiZkcrNMwt4qrSCz99wGcOHwOSWSLyqPdPKc5uOcMfcIjLTBudJ9rqjI6R98reLx9LQ3M5zm3S+JZFY9lRpBS3tQVYsHut3KVGlcPDJ3JJsZhZm8tjqg4SmXkQk1rR3BPn5O4e4akIuU0dn+F1OVCkcfGJmrFg8jrKqRlbv06m8RWLRq7uqOFJ3lhWLx/ldStQpHHx066wCcoYn89PVB/0uRUS68djqgxRmDeP6aSP9LiXqFA4+Sk0K8DcLSnh11wkqapr8LkdEwuw+fpp39p/iY1eNHdSn5u7J0PuJY8xHFpUQMOMnbx/0uxQRCfPoWwdISUzgQ/OHxhHR51M4+Kwgcxi3zR7Dk+sPU9fU6nc5IgKcON3Mc5uO8KEri8kenux3Ob5QOMSAe949gabWDn6hg+JEYsKjbx+gI+j4xNUT/C7FNwqHGDB1dAbvmZLPT94+qLO1ivjsdHMbv1xzmFtmFlCSm+Z3Ob5ROMSIe989kVNnWnlmQ6XfpYgMaU+sPUxDSzv3vnui36X4SuEQIxaOz2F2cRY//tN+OoI6KE7EDy3tHTzy1gGunpTHjMLBfRnQi1E4xAgz41PvnsChU028tP2Y3+WIDEm/2XSEqoYWPvnuoTvX0EnhEENumD6aCfnD+e/Xywlq60Ekqto7gnz/zX3MKMzg6kl5fpfjO4VDDAkkGJ9dMpndxxt0pTiRKHtu0xEOnWric0suGzLXbLgQhUOMuXXWGCbkD+c7r5Vp60EkSto6gvy/18uZUZjBkiF4qozuKBxiTPjWwx+09SASFc9tOsLhGm01hFM4xKBzWw+vautBJNJCWw1lzCzM1FZDGIVDDOrcethzooGXtmvrQSSSntt4hIqas3zu+snaagijcIhRt84aw6SRI/ivV/bQ3hH0uxyRQam5rYNvv7qXWUWZXDdVWw3hFA4xKpBg/PONU9hffYanSiv8LkdkUHps9UGO1jdz/81TtdVwHoVDDLth+ijmj83mW6+Ucaal3e9yRAaVuqZWvvdGOe+Zks/iiTqu4XwKhxhmZjxwyzRONrbw8J8O+F2OyKDy/Tf30dDSzv03T/W7lJikcIhx88Zmc/OM0Ty0ah/VDS1+lyMyKFTWNvHTtw/ywblFTB2d4Xc5MUnhEAe+eOMUmtuDfPvVvX6XIjIo/OfKPZjBF5Ze5ncpMUvhEAcm5I/gY4vG8sS6w+w4Wu93OSJxbf3BGn6z+Sh/f80ECjKH+V1OzFI4xInP33AZ2WnJPPj8DpzTgXEifdERdPzr8zsYk5nKp987tK/XcDEKhziROSyJf7lpKqWHavnN5iN+lyMSl3659hC7jp3my7dOJy050e9yYprCIY58cF4Rs4uz+N8v7qahuc3vckTiyqnGFr65cg+LJ+Zy84zRfpcT8xQOcSQhwfjqbZdzsrGF77xa5nc5InHlmyv30NTawf+67XId8NYLCoc4M7s4i+VXlvDo2wfYWlnndzkicWHN/lM8ub6Cv3vXOCaPSve7nLjQ53Aws2Ize8PMdpnZDjP7rNeeY2avmFmZ9z3bazcz+66ZlZvZVjObG/ZcK7z+ZWa2ov8/1uB2/81TyRuRwj8/s5U2nXdJ5IKa2zq4/9dbKclJ4ws3TPG7nLjRny2HduCfnHPTgEXAfWY2HbgfeM05Nxl4zbsPcDMw2fu6B/gBhMIEeBBYCCwAHuwMFOle5rAk/u32Gew+3sCP/rjP73JEYtq3Xt3LwVNNfP2OmQxLDvhdTtzoczg454455zZ6txuAXUAhsAx4zOv2GHC7d3sZ8DMXsgbIMrMC4EbgFedcjXOuFngFuKmvdQ0VSy8fzftmFfDd18opr2rwuxyRmLStsp4fr9rP8iuLWazrQl+SAZlzMLNxwBXAWmCUc+4YhAIE6DwPbiEQfnrRSq+tp3a5iK+8/3LSUgJ88ZmtOq23yHla2jv44jNbyBuRwgO3TPO7nLjT73AwsxHAr4HPOedOX6hrN23uAu3dvdY9ZlZqZqXV1dWXXuwgk5+ewleXzWDT4Tq+94aGl0TC/efKPew+3sB/3DGTzGFJfpcTd/oVDmaWRCgYHnfOPes1n/CGi/C+V3ntlUBx2MOLgKMXaO/COfeQc26+c25+fn5+f0ofNG6bPYbb54zhu6+XsfFwrd/liMSEt8pO8uM/HeCji0pYMm2U3+XEpf7srWTAI8Au59z/DVv0AtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0Uu9Numlr94+g9EZqXzuyc006roPMsTVnmnln361mYn5w/nSLdP9Lidu9WfL4V3Ax4DrzGyz93UL8HXgBjMrA27w7gO8COwHyoEfA58GcM7VAF8D1ntfX/XapJcyUpP49vI5VNY28ZUXdvhdjohvnHM88Ow2as608p3lV2jvpH7o88lFnHNv0f18AcCSbvo74L4enutR4NG+1iJw5bgcPvPeSXz39XIWjMvhr68svviDRAaZR98+yB92HOeBm6cyozDT73Limo6QHkQ+e/1lXD0pjy8/v53tR3Rqbxla1h+s4T9e3MUN00dxz7UT/C4n7ikcBpFAgvGd5XPIG57Mvb/YQF1Tq98liURFVUMz9z2+kaLsYfzXX8/WuZMGgMJhkMkdkcL3PjKXE6eb+dxTm+kI6toPMri1tgf5zC83cbq5jR98dB4ZqdptdSAoHAahK0qy+cptl/Pmnmr+7fc7/S5HJGKcc3zpuW2sO1DDNz4wi2kFuh70QNHVLgapjywcy76qMzz69gHG5w3nrqvG+V2SyID7/pv7+NWGSv5xyWSWzdGJFQaSwmEQ+9L7pnG4JrR7a3F2Gu+dOvLiDxKJE7/bepRvrtzDsjlj+Pz1k/0uZ9DRsNIg1jlBPa0gg/t+uZFNOoJaBol39p3iC09vYf7YbL7xgVmagI4AhcMgNzwlkUf/9kryRqTwtz9Zz+7jFzr9lUjs21xRxyceW8/YnDQeums+qUk60C0SFA5DwKiMVB7/xEKGJQX46MPrOHDyjN8lifTJ7uOnWfHoOnJHpPCLTywkZ3iy3yUNWgqHIaI4J41ffGIBQef46MNrOXyqye+SRC5JeVUDH3tkHalJCTz+iYWMykj1u6RBTeEwhEwamc7PPr6AM63t3Pmj1ZRXNfpdkkiv7Dhaz1//aA3OwS/uXkhxTprfJQ16CochZkZhJk/es4iOIHzoR++w86jmICS2bTxcy4cfWkNqYgK/uvcqJo9K97ukIUHhMARNHZ3B059cRHJiAssfeoe1+0/5XZJIt/64t5qPPbyW7OHJPH3vVYzPG+53SUOGwmGImpA/gqc/eRV56Sl87JF1/GbTEb9LEvkLv1x7mI//dD3FOWk8/cmrKMrWUFI0KRyGsOKcNJ791GKuKMnic09t5juvlhE6s7qIf4JBx3+8uIv/+dw2rpmcxzOfWqzJZx8oHIa4rLRkfn73Qu6YW8i3Xt3Lpx/fSENzm99lyRBVe6aVjz+2nh+t2s9HF5Xw8F3zGZGiEzn4QeEgJCcm8F93zuZLt0zj5Z0nuO2/39bBchJ1WyrquPX/vcXq8lN87fYZfG3ZDBID+ojyi9a8AGBm/P21E3ji7xdxpqWd27/3Nr9Yc0jDTBJxwaDj4T/t584fvgPAr+49NJXsAAAMlUlEQVS9io8tGqtTYvhM4SB/YcH4HH73j1dz5bgcvvyb7fztT9Zz4nSz32XJIFVZ28TfPLyGf/v9Lq69LI/f/cPVzC7O8rssASxe/zKcP3++Ky0t9buMQSsYdPxi7SH+94u7SEkM8OX3TeOD84r015wMiI6g44l1h/n6S7sB+Nf3T+dOvb8izsw2OOfm96qvwkEuZH91I198ZisbDtVy5bhs/u32mUwZrYOQpO+2H6nnS7/ZzpaKOhZPzOUbH5ilI56jROEgAyoYdPxqQwX/8dJuGpvb+eiisfzDdZPIHZHid2kSR6pON/OtV8t4av1hcoan8OX3TWPZnDHaWogihYNERM2ZVr65cg9Pl1YwLCnAJ6+dwN3XjCctWbsaSs8amtt4aNV+Hv7TAdqDQT6ycCyfv+EyMofpWs/RpnCQiCqvauT//GE3L+88QXZaEh9/13juumocmWn6zy5/VnOmlZ+8fYCfrj5IQ3M7t84q4Is3TmFsrk6B4ReFg0TFxsO1fP+Ncl7dVcWIlEQ+dGUxH1s0lnE6/82QVl7VwM/fOcTTpZU0t3dw0+Wjue+9k5hRmOl3aUOewkGiatex0/zwj/v4/dZjtAcd116Wz98sKOG9U/NJSdRVuoaC5rYOXt11gsfXHOad/adIDiRw25wx3PvuCUwaqR0YYoXCQXxRdbqZJ9ZV8Mt1hzhxuoXMYUncOquAZXMKmTc2m0CCJh4Hk/aOIOsO1PD85qO8uO0YDS3tFGYN4yOLSvjQ/GLtsBCDFA7iq/aOIH8qP8lzG4/w8s7jNLcFyR2ezPXTRnH99FEsmpBDeqrmJ+JR/dk23tl3kld2VvHa7hPUNbWRlhzgphmjueOKIq6amKs/AmLYpYSDdjORAZcYSOC9U0by3ikjaWxp543dVby88wS/33aMp0orSEwwrijJYvHEPK4cl8OckiydXC1GnW5uY9PhOkoP1vBW+Um2VNQRdJCRmsiSaaNYOn0U756Srz3WBqGY2XIws5uA7wAB4GHn3Ncv1F9bDvGnpb2DDYdqeavsJG+Vn2TbkXqcgwSDKaMzmFmYwYzCTKYXZDB5VLp2dYyyuqZW9p5oZMfRenYcPc32I/XsOdFw7nc0qyiLaybncfWkPOaOzSZJJ8WLO3E3rGRmAWAvcANQCawHPuyc29nTYxQO8a+h86/SQ7VsOlzLjqOnqTnTem75yPQUJo0cwdjc4ZTkpDE2N42CzFQKMoeRn56i4YtL1N4RpLqxhWP1zRytO8vhmiYOn2ri4KkzlFed4WRjy7m+eSNSmFGYwRXF2cwfl82c4iyGa+su7sXjsNICoNw5tx/AzJ4ElgE9hoPEv/TUJK69LJ9rL8sHwDnH8dPN7Dx6mvKqRsqqGtlX3cjLO45zKiw0AAIJRu7wZPJGpJCXnkJ2WhLZaclkDksiPTWRjNTQ97SURIYnBxiWHGBYUoCUpACpiQkkJyaQFEggOZBAQoyHTDDoaO0I0tYRpKU99NXc1sHZ1g7OtnXQ1NrBmZZ2GprbaGhu53RzO/VNrdQ2tVHb1MrJxlaqG1qoOdNC8Ly/BfNGJFOSk8Z1U/OZPDKdSSNHMH1Mhi6uIzETDoVARdj9SmChT7WIT8yMgsxhFGQOY8m0UX+xrKG5jYqasxyrP8ux+maO1zdT3dDCycbQ18GTZ6hrauV0c/slv24gwUhMMJICCQQSjECCkWBGIAESLHQ7IQEMI8FCdRqAlynm1R7WROdnsHPu3G1cqN05R9CBwxEMQtA5gs7R4d1u7wjSEXS0e18d53+i90LmsCSyvMAszBrG7KJM8tNTGJ2ZypjMYYzOTKUkJ01bA9KjWHlndPenW5f/EWZ2D3APQElJSaRrkhiSnprE9DFJTB+TccF+7R1BzrR0cNr7K/psWztnWkJ/Xbe0d9Dc1kFzW5DW9iCtHaHv7cEg7R3eh3FHkI7OD+qg8z64Qx/anR/0Qce561y4c/+EPuzDWVh6dL7BzbyAOXc7dD8UQKGQ6gyoxMCfQyspkECKt8WTmhggJSmB1KQAacmdX4mkpyaSnpLEiNREDblJv8VKOFQCxWH3i4Cj53dyzj0EPAShOYfolCbxJDGQQGZagk7lIdJPsbK7wXpgspmNN7NkYDnwgs81iYgMWTGx5eCcazezzwArCe3K+qhzbofPZYmIDFkxEQ4AzrkXgRf9rkNERGJnWElERGKIwkFERLpQOIiISBcKBxER6ULhICIiXcTEiff6wsyqgUN9fHgecHIAyxkoquvSqK5Lo7ouzWCsa6xzLr83HeM2HPrDzEp7e2bCaFJdl0Z1XRrVdWmGel0aVhIRkS4UDiIi0sVQDYeH/C6gB6rr0qiuS6O6Ls2QrmtIzjmIiMiFDdUtBxERuYBBGw5mdqeZ7TCzoJnNP2/ZA2ZWbmZ7zOzGHh4/3szWmlmZmT3lnUp8oGt8ysw2e18HzWxzD/0Omtk2r1/EL5xtZl8xsyNhtd3SQ7+bvHVYbmb3R6Gub5rZbjPbambPmVlWD/2isr4u9vObWYr3Oy733kvjIlVL2GsWm9kbZrbLe/9/tps+7zGz+rDf779Gui7vdS/4e7GQ73rra6uZzY1CTVPC1sNmMzttZp87r09U1peZPWpmVWa2Pawtx8xe8T6HXjGz7B4eu8LrU2ZmKwakINd5hatB9gVMA6YAbwLzw9qnA1uAFGA8sA8IdPP4p4Hl3u0fAp+KcL3/BfxrD8sOAnlRXHdfAf7HRfoEvHU3AUj21un0CNe1FEj0bn8D+IZf66s3Pz/waeCH3u3lwFNR+N0VAHO92+nA3m7qeg/wu2i9n3r7ewFuAV4idKG8RcDaKNcXAI4TOhYg6usLuBaYC2wPa/s/wP3e7fu7e88DOcB+73u2dzu7v/UM2i0H59wu59yebhYtA550zrU45w4A5cCC8A4WuiDwdcAzXtNjwO2RqtV7vb8GnojUa0TAAqDcObffOdcKPElo3UaMc+5l51znRaLXELpioF968/MvI/TegdB7aYl1Xmw6Qpxzx5xzG73bDcAuQtdojwfLgJ+5kDVAlpkVRPH1lwD7nHN9Pbi2X5xzq4Ca85rD30M9fQ7dCLzinKtxztUCrwA39beeQRsOF1AIVITdr6Trf55coC7sg6i7PgPpGuCEc66sh+UOeNnMNnjX0Y6Gz3ib9o/2sCnbm/UYSR8n9Fdmd6Kxvnrz85/r472X6gm9t6LCG8a6AljbzeKrzGyLmb1kZpdHqaSL/V78fk8tp+c/0PxYXwCjnHPHIBT8wMhu+kRkvcXMxX76wsxeBUZ3s+hLzrnne3pYN23n77LVmz690ssaP8yFtxre5Zw7amYjgVfMbLf3V0afXagu4AfA1wj9zF8jNOT18fOfopvH9nvXt96sLzP7EtAOPN7D0wz4+uqu1G7aIvY+ulRmNgL4NfA559zp8xZvJDR00ujNJ/0GmByFsi72e/FzfSUDtwEPdLPYr/XVWxFZb3EdDs656/vwsEqgOOx+EXD0vD4nCW3SJnp/8XXXZ0BqNLNE4A5g3gWe46j3vcrMniM0pNGvD7verjsz+zHwu24W9WY9Dnhd3mTbrcAS5w24dvMcA76+utGbn7+zT6X3e86k67DBgDOzJELB8Lhz7tnzl4eHhXPuRTP7vpnlOecieh6hXvxeIvKe6qWbgY3OuRPnL/BrfXlOmFmBc+6YN8RW1U2fSkLzIp2KCM219stQHFZ6AVju7UkyntBfAOvCO3gfOm8AH/SaVgA9bYn01/XAbudcZXcLzWy4maV33iY0Kbu9u74D5bxx3r/q4fXWA5MttFdXMqFN8hciXNdNwL8AtznnmnroE6311Zuf/wVC7x0IvZde7ynQBoo3p/EIsMs593976DO6c+7DzBYQ+hw4FeG6evN7eQG4y9traRFQ3zmkEgU9br37sb7ChL+HevocWgksNbNsbwh4qdfWP5Gegffri9CHWiXQApwAVoYt+xKhPU32ADeHtb8IjPFuTyAUGuXAr4CUCNX5U+De89rGAC+G1bHF+9pBaHgl0uvu58A2YKv35iw4vy7v/i2E9obZF6W6ygmNrW72vn54fl3RXF/d/fzAVwmFF0Cq994p995LE6Kwjq4mNKSwNWw93QLc2/k+Az7jrZsthCb2F0ehrm5/L+fVZcD3vPW5jbC9DCNcWxqhD/vMsLaory9C4XQMaPM+u+4mNEf1GlDmfc/x+s4HHg577Me991k58HcDUY+OkBYRkS6G4rCSiIhchMJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLv4/Uw5YHte+pP0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# constant noise on std\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "N=100\n",
    "ratio=0.1\n",
    "num_itr=50\n",
    "std_noise = 0.1\n",
    "\n",
    "f=lambda x: 3*x**2+x**4  # function of f(x)\n",
    "samples = np.random.normal(0,1,N) # x-value of f(x)\n",
    "print('true min: ', np.min(f(samples)))\n",
    "for i in range(num_itr):\n",
    "    # sample and select top 10% (max [-a:]or min [:a])\n",
    "    selected_samples = samples[np.array(f(samples)).argsort()[:int(N*ratio)]]\n",
    "    m=np.mean(selected_samples)\n",
    "    std=np.std(selected_samples)+std_noise  # constant noise on std\n",
    "#     print(std)\n",
    "    # fit the gaussian\n",
    "    samples=np.random.normal(m, std, N)\n",
    "    print(f(selected_samples))\n",
    "\n",
    "'''display the function f(x) '''\n",
    "x=np.linspace(-10,10,N)\n",
    "y=f(x)\n",
    "plt.plot(x,y)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('true min: ', 9.848678761433567e-05)\n",
      "0.16098619064378178\n",
      "[9.84867876e-05 2.05477238e-04 1.16920654e-03 2.05328101e-03\n",
      " 5.52646690e-03 5.77471848e-03 1.47582847e-02 1.58134710e-02\n",
      " 3.55514899e-02 4.06655378e-02]\n",
      "0.11460017007264803\n",
      "[1.52200969e-06 1.81824287e-04 4.30198218e-04 1.02625256e-03\n",
      " 1.09172702e-03 1.16851665e-03 1.36434360e-03 1.50257194e-03\n",
      " 1.59505580e-03 1.71016474e-03]\n",
      "0.10316136007950855\n",
      "[2.78924072e-06 2.92469716e-06 4.57427452e-06 1.15947573e-05\n",
      " 5.88072233e-05 1.81596884e-04 2.02885092e-04 2.14681729e-04\n",
      " 4.90750063e-04 5.47463261e-04]\n",
      "0.10097706158161131\n",
      "[1.05558571e-05 1.08541522e-05 3.87355124e-05 5.96325516e-05\n",
      " 1.09972605e-04 1.75154079e-04 1.95055772e-04 2.14747638e-04\n",
      " 3.41215753e-04 4.17070325e-04]\n",
      "0.09911446576625192\n",
      "[2.67139142e-06 8.22362352e-06 2.08395990e-05 6.59411849e-05\n",
      " 8.88284682e-05 1.52864573e-04 2.05977949e-04 2.58289207e-04\n",
      " 3.72801780e-04 4.08436004e-04]\n",
      "0.09405478058787772\n",
      "[8.36932167e-07 1.79258593e-05 1.87011384e-05 3.25031282e-05\n",
      " 6.55060560e-05 6.69242711e-05 9.67320765e-05 1.01577451e-04\n",
      " 1.10910464e-04 1.34662548e-04]\n",
      "0.09581568026834013\n",
      "[1.87671609e-08 6.84977613e-05 6.87092302e-05 1.77648188e-04\n",
      " 1.78960621e-04 2.07951824e-04 2.08749621e-04 2.32087932e-04\n",
      " 3.41137939e-04 4.45461941e-04]\n",
      "0.09557669369577773\n",
      "[7.18186961e-08 1.19025764e-06 3.09385944e-04 3.09553882e-04\n",
      " 4.47140042e-04 4.48996212e-04 4.52010964e-04 4.89406319e-04\n",
      " 4.89540186e-04 5.46414262e-04]\n",
      "0.08864849127900984\n",
      "[3.73326341e-06 1.21682380e-05 1.84627076e-05 6.73294984e-05\n",
      " 7.50381941e-05 8.95725494e-05 1.00058797e-04 1.03227004e-04\n",
      " 1.17377009e-04 1.47474384e-04]\n",
      "0.0909038174834493\n",
      "[2.31514275e-07 8.43977990e-06 4.95918994e-05 8.93150704e-05\n",
      " 1.81092012e-04 2.55368557e-04 2.88407781e-04 3.96129155e-04\n",
      " 5.63251539e-04 5.64110042e-04]\n",
      "0.08786160002169964\n",
      "[6.88160438e-06 2.12016102e-05 2.43148853e-05 6.16166896e-05\n",
      " 1.61943419e-04 2.68603266e-04 3.35619745e-04 3.46577562e-04\n",
      " 3.80886788e-04 4.03665279e-04]\n",
      "0.08303961625914558\n",
      "[2.14863437e-08 6.60497846e-07 9.02760470e-07 9.23794061e-06\n",
      " 3.98465005e-05 5.38152805e-05 1.52244444e-04 1.54458844e-04\n",
      " 1.61774284e-04 3.07417189e-04]\n",
      "0.08126387389018269\n",
      "[2.39097426e-05 2.72829643e-05 4.24513565e-05 5.77412147e-05\n",
      " 6.22797820e-05 8.28087484e-05 8.77957692e-05 1.35707262e-04\n",
      " 1.85545821e-04 2.34252947e-04]\n",
      "0.07874478703068684\n",
      "[2.87886072e-05 4.33074263e-05 4.35404845e-05 1.32899458e-04\n",
      " 1.49292001e-04 1.68472087e-04 1.95445446e-04 2.15385373e-04\n",
      " 2.60747094e-04 3.25172977e-04]\n",
      "0.08018618296902304\n",
      "[1.38215581e-06 2.65796529e-06 3.13170024e-06 2.53304125e-05\n",
      " 3.65983624e-05 1.14774321e-04 2.45086542e-04 2.97204843e-04\n",
      " 6.50594264e-04 6.90538840e-04]\n",
      "0.07285875055297013\n",
      "[2.58580471e-07 4.77473496e-07 8.07179917e-07 1.13782078e-06\n",
      " 4.77915507e-06 2.61974978e-05 5.41006306e-05 7.25787433e-05\n",
      " 7.47883285e-05 8.49246080e-05]\n",
      "0.07494860782100811\n",
      "[1.37315450e-09 1.50638550e-07 2.58396469e-06 9.54457020e-05\n",
      " 1.38570300e-04 2.46604369e-04 2.58710948e-04 3.07393323e-04\n",
      " 3.09390011e-04 3.42513106e-04]\n",
      "0.07135772572655984\n",
      "[5.08217477e-07 8.18468493e-07 5.23644326e-06 9.52091535e-06\n",
      " 1.50450220e-05 4.44904233e-05 1.55386639e-04 1.80280358e-04\n",
      " 2.05052554e-04 3.13443995e-04]\n",
      "0.0694518726950622\n",
      "[6.85642484e-07 5.04721769e-06 1.37625988e-05 1.57253182e-05\n",
      " 3.37040747e-05 7.00715942e-05 9.76361373e-05 1.54480683e-04\n",
      " 1.60223714e-04 3.45371453e-04]\n",
      "0.06764603941459735\n",
      "[1.01382346e-06 1.06858616e-06 4.13445790e-06 8.87494608e-05\n",
      " 8.98343322e-05 1.15969965e-04 1.18110686e-04 1.22441403e-04\n",
      " 1.59009467e-04 2.64838411e-04]\n",
      "0.06529104431371574\n",
      "[2.99110005e-06 6.29825933e-06 7.49588151e-06 2.76204900e-05\n",
      " 4.36600029e-05 5.68493360e-05 1.16097098e-04 1.50787776e-04\n",
      " 2.14172389e-04 2.20287203e-04]\n",
      "0.06290965201559266\n",
      "[1.46717425e-11 3.14722298e-07 2.28180921e-05 2.55369635e-05\n",
      " 4.15963147e-05 8.41446327e-05 8.79076340e-05 1.40333239e-04\n",
      " 1.58758650e-04 1.61858541e-04]\n",
      "0.06128571779545415\n",
      "[1.99861586e-05 3.30288003e-05 5.18063961e-05 6.72794790e-05\n",
      " 7.20022816e-05 8.78552888e-05 9.75173014e-05 1.29750744e-04\n",
      " 1.42781490e-04 1.68589221e-04]\n",
      "0.06040350452373543\n",
      "[1.19876914e-06 1.02804478e-05 2.58089746e-05 3.52414947e-05\n",
      " 5.68443221e-05 6.43794635e-05 7.64483158e-05 2.40777720e-04\n",
      " 3.51474269e-04 3.72823880e-04]\n",
      "0.05657374798662648\n",
      "[7.52908413e-06 8.74773893e-06 2.42018498e-05 4.37637887e-05\n",
      " 5.01205457e-05 5.80240861e-05 5.88367982e-05 8.19357529e-05\n",
      " 1.50424231e-04 1.54169126e-04]\n",
      "0.05311100197674016\n",
      "[8.37695172e-08 1.50219756e-07 2.38538245e-06 3.22646379e-06\n",
      " 1.09111734e-05 3.23810256e-05 3.28755669e-05 9.24835917e-05\n",
      " 1.07545810e-04 1.32797577e-04]\n",
      "0.05076430322299557\n",
      "[6.04417945e-08 1.90854604e-06 1.97339459e-06 6.87533011e-06\n",
      " 1.17718609e-05 1.21438521e-05 1.34687509e-05 1.81939467e-05\n",
      " 7.43120997e-05 8.98695566e-05]\n",
      "0.049873023935417996\n",
      "[5.18976275e-06 5.77087301e-06 1.01047848e-05 1.99755114e-05\n",
      " 2.55513850e-05 3.07053205e-05 6.15222641e-05 6.30285355e-05\n",
      " 7.47215567e-05 1.87123473e-04]\n",
      "0.048581861449056574\n",
      "[9.85732422e-07 6.09958622e-06 1.70369651e-05 3.04801771e-05\n",
      " 5.91212671e-05 7.35344256e-05 8.58397473e-05 9.13898170e-05\n",
      " 1.07997207e-04 1.65366972e-04]\n",
      "0.04477445612660895\n",
      "[5.86725223e-10 6.41865442e-10 2.35421533e-09 7.52506913e-07\n",
      " 1.38253554e-05 1.70545320e-05 2.69874120e-05 3.41196345e-05\n",
      " 3.58981683e-05 1.28453920e-04]\n",
      "0.04508255640501922\n",
      "[5.57240692e-08 5.31577303e-06 7.01170239e-06 1.53904913e-05\n",
      " 2.69219648e-05 7.85777709e-05 1.25494404e-04 1.48136530e-04\n",
      " 1.69769315e-04 2.06468236e-04]\n",
      "0.04117454860880181\n",
      "[1.01089793e-06 3.92009725e-06 9.15621312e-06 1.37725219e-05\n",
      " 2.47392328e-05 2.75008739e-05 3.24036503e-05 4.01790927e-05\n",
      " 7.02277066e-05 7.96190446e-05]\n",
      "0.03864170449708435\n",
      "[2.41774110e-06 4.56141932e-06 4.64516916e-06 8.35676861e-06\n",
      " 8.86471190e-06 1.93382885e-05 3.48933417e-05 4.02273641e-05\n",
      " 4.13731571e-05 4.46823108e-05]\n",
      "0.035982079404857516\n",
      "[4.98719659e-06 7.91241601e-06 1.25322809e-05 1.77546232e-05\n",
      " 1.88286956e-05 2.28939972e-05 2.67429246e-05 3.00927326e-05\n",
      " 3.31214574e-05 4.75830474e-05]\n",
      "0.034663790466676024\n",
      "[3.45428466e-09 1.70193187e-07 3.77691070e-07 1.42006531e-05\n",
      " 1.49111036e-05 1.81117311e-05 3.16518777e-05 3.80854572e-05\n",
      " 4.85375761e-05 4.98411655e-05]\n",
      "0.031906662136658125\n",
      "[4.76203052e-07 2.53260457e-06 3.08020527e-06 3.13662101e-06\n",
      " 3.94836218e-06 9.61971458e-06 1.27633763e-05 1.92569543e-05\n",
      " 2.55102096e-05 2.92824970e-05]\n",
      "0.030834043618010207\n",
      "[2.96803708e-07 3.50695333e-07 9.56528666e-07 2.29956958e-05\n",
      " 2.76189196e-05 4.08145553e-05 5.76459070e-05 7.20421537e-05\n",
      " 7.79246782e-05 7.80296218e-05]\n",
      "0.028484534163076915\n",
      "[2.55210426e-07 8.49208655e-07 1.27173235e-06 3.75865281e-06\n",
      " 6.46614706e-06 2.30770047e-05 2.66568689e-05 3.20342841e-05\n",
      " 4.28596560e-05 4.79996413e-05]\n",
      "0.026942185458307975\n",
      "[1.05019782e-06 5.94699251e-06 6.63960947e-06 2.19036545e-05\n",
      " 2.21948110e-05 2.27453348e-05 2.78261462e-05 3.67160294e-05\n",
      " 5.67527378e-05 6.65330102e-05]\n",
      "0.024930337978211023\n",
      "[5.42639765e-06 6.42038745e-06 7.68260429e-06 7.97164346e-06\n",
      " 1.86816867e-05 2.28346835e-05 2.50569131e-05 4.42919581e-05\n",
      " 5.15285422e-05 6.98153762e-05]\n",
      "0.021885706063445663\n",
      "[2.29729952e-06 2.87027673e-06 4.31507462e-06 5.36572107e-06\n",
      " 9.82768730e-06 1.03457748e-05 1.32506737e-05 1.55200679e-05\n",
      " 1.73241674e-05 2.89793146e-05]\n",
      "0.019212552096214426\n",
      "[7.72715265e-08 2.20343133e-07 2.27116186e-07 2.59107348e-06\n",
      " 3.78692388e-06 4.63136899e-06 4.92737395e-06 9.13563747e-06\n",
      " 1.38083142e-05 1.59908874e-05]\n",
      "0.017306856797596616\n",
      "[1.65963105e-07 1.53775102e-06 2.70062010e-06 2.87661815e-06\n",
      " 3.06738018e-06 3.51464393e-06 4.54749147e-06 1.08952731e-05\n",
      " 1.21106249e-05 1.49817999e-05]\n",
      "0.01528153902464059\n",
      "[8.39366803e-08 8.88673078e-08 2.84847038e-06 3.99861098e-06\n",
      " 4.30063411e-06 7.44063545e-06 7.49987647e-06 9.84848353e-06\n",
      " 1.24202857e-05 1.85789708e-05]\n",
      "0.013265478401293725\n",
      "[2.13228849e-08 3.11633994e-08 3.18932928e-07 7.44276450e-07\n",
      " 1.05658285e-06 6.38964335e-06 1.78447079e-05 1.80701180e-05\n",
      " 1.91075959e-05 1.96189600e-05]\n",
      "0.010628498297252346\n",
      "[1.13222901e-07 3.05851290e-07 3.52011883e-07 4.57306971e-07\n",
      " 5.35431298e-07 1.13568686e-06 2.01380855e-06 2.29939282e-06\n",
      " 2.49151929e-06 4.12443729e-06]\n",
      "0.00846528907245487\n",
      "[6.25285398e-08 1.12805767e-07 2.92049591e-07 3.83934691e-07\n",
      " 4.37454884e-07 6.12138948e-07 9.63057082e-07 1.09200314e-06\n",
      " 1.19295580e-06 1.34612406e-06]\n",
      "0.006370729356429941\n",
      "[1.64767303e-08 2.21014478e-08 2.81135488e-08 3.97282592e-08\n",
      " 1.14540851e-07 6.83879513e-07 8.32273995e-07 9.25609640e-07\n",
      " 1.03896371e-06 1.26552152e-06]\n",
      "0.004440967446293719\n",
      "[1.22648043e-08 3.90619756e-08 5.94507382e-08 4.93842145e-07\n",
      " 5.50362811e-07 7.71518782e-07 8.27987645e-07 8.37205190e-07\n",
      " 1.06773552e-06 1.19328596e-06]\n",
      "0.0022634864932339167\n",
      "[7.28047703e-10 1.04692437e-08 6.07451820e-08 7.16425535e-08\n",
      " 7.94342192e-08 2.03501224e-07 3.28163494e-07 4.10507604e-07\n",
      " 4.70225814e-07 4.99707346e-07]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8XGed7/HPT6Nm2eqSbVnFPS5xi+3YjkkCxIlTCHE2kMUsEO8SNgTCLmUvu8mF14YLu3fhsnspd2khCQQIKYSEBEhwOiZxXOTeLblKbpKtYsmy6jz3jzkygyXZsqSZMyN936+XrJnnPDPz09F4vjrPc4o55xAREQmX4HcBIiISexQOIiLShcJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLhL9LqCv8vLy3Lhx4/wuQ0QkbmzYsOGkcy6/N33jNhzGjRtHaWmp32WIiMQNMzvU274aVhIRkS4UDiIi0oXCQUREulA4iIhIFwoHERHpQuEgIiJdKBxERKSLIRUOre1BfvDmPlbtrfa7FBGRS/bqzhM8tGof0bi885AKh6SA8dCqffx+6zG/SxERuWTPbTrCz9ccwswi/lpDKhzMjBmFmWw7Uu93KSIil2zbkXpmFmZG5bUuGg5m9qiZVZnZ9rC2HDN7xczKvO/ZXruZ2XfNrNzMtprZ3LDHrPD6l5nZirD2eWa2zXvMdy3CkTirKJO9JxpobuuI5MuIiAyouqZWDtc0MbMwKyqv15sth58CN53Xdj/wmnNuMvCadx/gZmCy93UP8AMIhQnwILAQWAA82BkoXp97wh53/msNqJmFmbQHHbuPN0TyZUREBtT2I6cBYmfLwTm3Cqg5r3kZ8Jh3+zHg9rD2n7mQNUCWmRUANwKvOOdqnHO1wCvATd6yDOfcOy40w/KzsOeKiJlFodTdVlkXyZcRERlQW4+EPrNiJhx6MMo5dwzA+z7Say8EKsL6VXptF2qv7KY9YsZkppIzPFnzDiISV7YfqackJ43MtKSovN5AT0h3N1/g+tDe/ZOb3WNmpWZWWl3dt91RzYyZhZlsrVQ4iEj82FpZz8yi6Gw1QN/D4YQ3JIT3vcprrwSKw/oVAUcv0l7UTXu3nHMPOefmO+fm5+f36noV3ZpZmElZVaMmpUUkLtSeaaWy9mzUhpSg7+HwAtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0UuBld6yBjNb5O2ldFfYc0XMzKJMOoKOncdOR/qlRET6rXMYfFYshYOZPQG8A0wxs0ozuxv4OnCDmZUBN3j3AV4E9gPlwI+BTwM452qArwHrva+vem0AnwIe9h6zD3hpYH60nnWm73bNO4hIHOgMh8ujGA4XvUyoc+7DPSxa0k1fB9zXw/M8CjzaTXspMONidQykgsxU8kYka95BROLCtsp6xuWmkTksOpPRMMSOkO7UeaS0thxEJB5sO1LPjChuNcAQDQcIjd3tPdHA2VZNSotI7DrV2MKRurPMiuKeSjCEw2FGYSZBBzuPaetBRGJX53yDthyiZNa5I6UVDiISu7YrHKJrVEYK+ekpmpQWkZi2tbKe8XnDyUiN3mQ0DOFwMDNmF2WyWedYEpEY5Zxjc0Uds6M83wBDOBwA5hRnsb/6DPVNbX6XIiLSxbH6ZqoaWphTHJ3TdIcb4uEQOmv4Fm09iEgM2lIR+myaU5J9kZ4Db0iHw6ziTMz+/AsQEYklmyvqSA4kMK0gPeqvPaTDISM1iYn5I9iscBCRGLSpoo5pYzJISQxE/bWHdDhAaN5hc0UdoTN/iIjEhvaOINsq67nCh/kGUDgwuziLU97pcEVEYkVZVSNn2zp8mYwGhcO5VN6koSURiSGdw90KB59MGZ1OSmKCJqVFJKZsPlxHVloSY3PTfHn9IR8OSYEEZhZmalJaRGJK6OC3LELXQYu+IR8OENps236knraOoN+liIjQ2NLO3qoG34aUQOEAhCalW9qD7D7W4HcpIiJsq6zHOZhTonDwVWc6b66o9bkSEZE/T0bPLlI4+Kooexh5I5K1x5KIxITNFbWMzU0jZ3iybzUoHAidofWKkmw2HVY4iIi/nHNsPFzn28FvnRQOnnljszlw8gynGlv8LkVEhrDK2rNUN7Qwb1yOr3UoHDzzxobOerhRWw8i4qPSQzUAzPPhTKzhFA6emYWZJAXs3C9GRMQPGw7VMiIlkSmjo38m1nAKB09qUoAZhZlsPKQ9lkTEPxsO1XFFSRaBBH8OfuukcAgzrySbLZX1tLbrYDgRib6G5jb2HD/NXJ+HlEDh8Bfmjc2mtT3IjqP1fpciIkPQ5oo6gg7mj1M4xJTOSekNGloSER9sOFSLmX9nYg2ncAgzMiOV4pxhCgcR8cWGQ7VMGZVOemqS36UoHM43rySb0kO1ujKciERVR9Cx6XBdTAwpgcKhi3ljs6luaNGV4UQkqvaeaKCxpf3c8Lbf+hUOZvZ5M9thZtvN7AkzSzWz8Wa21szKzOwpM0v2+qZ498u95ePCnucBr32Pmd3Yvx+pf+aNDR2VqKElEYmmUu8zZ16Jv0dGd+pzOJhZIfCPwHzn3AwgACwHvgF8yzk3GagF7vYecjdQ65ybBHzL64eZTfcedzlwE/B9Mwv0ta7+mjI6neHJAYWDiETVxkO15KenUJwzzO9SgP4PKyUCw8wsEUgDjgHXAc94yx8DbvduL/Pu4y1fYqFLHC0DnnTOtTjnDgDlwIJ+1tVngYTQSfjWH9SR0iISPesP1jCvJNu3K7+dr8/h4Jw7AvwncJhQKNQDG4A651y7160SKPRuFwIV3mPbvf654e3dPOYvmNk9ZlZqZqXV1dV9Lf2iFozPYc+JBuqaWiP2GiIinY7UnaWy9iwLxsfGkBL0b1gpm9Bf/eOBMcBw4OZuunbu9tNdHLoLtHdtdO4h59x859z8/Pz8Sy+6lxaMz8E5KD2ooSURibz1B0IjFYMiHIDrgQPOuWrnXBvwLLAYyPKGmQCKgKPe7UqgGMBbngnUhLd38xhfzCnOIjmQwDoNLYlIFKw9UEN6aiLTCjL8LuWc/oTDYWCRmaV5cwdLgJ3AG8AHvT4rgOe92y949/GWv+5CBxO8ACz39mYaD0wG1vWjrn5LTQowpziLtQcUDiISeesOnOLKcTm+n2wvXH/mHNYSmljeCGzznush4F+AL5hZOaE5hUe8hzwC5HrtXwDu955nB/A0oWD5A3Cfc66jr3UNlAXjc9h+pJ4zLe0X7ywi0kcnG1vYV30mpoaUILS3UZ855x4EHjyveT/d7G3knGsG7uzhef4d+Pf+1DLQFozP4b/fKGfj4VqumRy5+Q0RGdpicb4BdIR0j+aOzSaQYKzT0JKIRNDaAzUMSwowY0ym36X8BYVDD0akJDJjTAZr9yscRCRy1h6oYe7YLJITY+vjOLaqiTELJ+SyuaKO5jbfp0BEZBCqb2pj9/HTLByf63cpXSgcLmDBuBxaO4JsqajzuxQRGYRKD9XgXOzNN4DC4YKuHJeDGZp3EJGIWHeghuRAQkxc3Od8CocLyExLYsqodB3vICIRsfZADbOLM0lN8u1coz1SOFzEogm5lB6qobU96HcpIjKINDS3se1IfUzON4DC4aIWT8yluS3IZs07iMgAWn+who6gY/EkhUNcWjghlwSDt8tP+l2KiAwib5efIjkxgbklsXHlt/MpHC4ic1gSMwozeWffKb9LEZFBZPW+U8wfmx2T8w2gcOiVxRPz2FRRS1OrzrMkIv1Xc6aVXcdO865JeX6X0iOFQy8snphLW4djva7vICIDoHMk4qqJsTnfAAqHXpk/LpukgLF6n+YdRKT/Vu87yYiURGYVxtb5lMIpHHohLTmRK0qyWV2ueQcR6b/V+06xcHwOiYHY/QiO3cpizOKJuWw/Wk99U5vfpYhIHDtad5YDJ8/E9JASKBx6bfHEPJyDNQe09SAifdc537B4YuxORoPCodfmFGcxLCnAah3vICL98Pa+k+QMT2bq6HS/S7kghUMvJScmcOX4HN7W8Q4i0kfOOd7Zd4qrJuSSEEPXi+6OwuESXD0pl/KqRo7Vn/W7FBGJQ/uqGzlW3xzTxzd0UjhcgmsvC11L+k97NbQkIpfuj95nx7WXKRwGlSmj0hmVkcIfy6r9LkVE4tCqvdVMyB9OUXaa36VclMLhEpgZ10zO562yk3QEnd/liEgcaW7rYO2BU1w7Od/vUnpF4XCJrr0sn/qzbWyt1Cm8RaT31h+sobktyLsvUzgMSldPysMMVmneQUQuwaq91SQHElg4IfauF90dhcMlyhmezMzCTFZp3kFELsGqvSeZPy6btOREv0vpFYVDH1w7OZ/NFXXUn9WpNETk4o7XN7PnRMO5PR7jgcKhD669LJ+OoNPR0iLSK50jDfEyGQ0Khz65oiSLESmJrCpTOIjIxf2p7CT56SlMK4jtU2aEUzj0QVIggcUTc1m1txrntEuriPSsI+h4q6yaaybnYRbbp8wI169wMLMsM3vGzHab2S4zu8rMcszsFTMr875ne33NzL5rZuVmttXM5oY9zwqvf5mZrejvDxUN75kykiN1ZymvavS7FBGJYVsq66htaoubXVg79XfL4TvAH5xzU4HZwC7gfuA159xk4DXvPsDNwGTv6x7gBwBmlgM8CCwEFgAPdgZKLLtu6kgAXttd5XMlIhLLXt9VRSDBeM9lI/0u5ZL0ORzMLAO4FngEwDnX6pyrA5YBj3ndHgNu924vA37mQtYAWWZWANwIvOKcq3HO1QKvADf1ta5oGZ2ZyuVjMnh9l8JBRHr22u4q5o/NJjMtye9SLkl/thwmANXAT8xsk5k9bGbDgVHOuWMA3vfOuCwEKsIeX+m19dQe866bOpLSQzXUNbX6XYqIxKCjdWfZdez0uZGGeNKfcEgE5gI/cM5dAZzhz0NI3eluJsZdoL3rE5jdY2alZlZaXe3/QWjXTR1J0MEf9/pfi4jEnte9Yecl04ZWOFQClc65td79ZwiFxQlvuAjve1VY/+KwxxcBRy/Q3oVz7iHn3Hzn3Pz8fP8nd2YXZZE7PJnXNLQkIt14fXcVJTlpTMwf4Xcpl6zP4eCcOw5UmNkUr2kJsBN4Aejc42gF8Lx3+wXgLm+vpUVAvTfstBJYambZ3kT0Uq8t5iUkGO+dOpI391TR3hH0uxwRiSFnWzt4u/wk100dGVe7sHbq70k+/gF43MySgf3A3xEKnKfN7G7gMHCn1/dF4BagHGjy+uKcqzGzrwHrvX5fdc7V9LOuqFkydSTPbKhkw6FaFk7I9bscEYkRq/edpKU9GJdDStDPcHDObQbmd7NoSTd9HXBfD8/zKPBof2rxy9WT80gKGK/vrlI4iMg5r+2uYnhygAXj4+MsrOfTEdL9lJ6axMLxuecmnkREnHO8sbuKaybnk5IY8LucPlE4DIDrpo6krKqRgyfP+F2KiMSAHUdPc6y+mevidEgJFA4D4obpowBYueO4z5WISCxYueM4CRaak4xXCocBUJyTxozCDIWDiAChcFgwPofcESl+l9JnCocBcuP00Ww8XMeJ081+lyIiPtpf3cjeE43cePlov0vpF4XDALlxRuiN8PLOEz5XIiJ+Wrkj9BmwVOEgAJNHjmBC3nBe1tCSyJC2csdxZhVlUpg1zO9S+kXhMEDMjKWXj+adfaeob9K1pUWGouP1zWyuqIv7ISVQOAyoGy8fRXvQ8dpuDS2JDEUv7wyNHNx4+SifK+k/hcMAml2UxeiMVO21JDJErdxxnIn5w5k0Mn6uFd0ThcMASkgwll4+ij/uraaptd3vckQkiuqaWlmzv2ZQDCmBwmHA3TRjNM1tQd7YrWs8iAwlK3ccpyPouHlGgd+lDAiFwwBbOD6X/PQUfrul20tSiMgg9dstxxibGzogdjBQOAywQIJxy4zRvLGnisYWDS2JDAUnG1tYve8kt84qiMtrN3RH4RAB7589hpb2IK/qgDiRIeGl7ccJutD//cFC4RABc0uyKchM1dCSyBDx2y1HmTRyBFNGxf9eSp0UDhGQkGC8b2YBq8qqdUCcyCB34nQz6w/WDKohJVA4RMz7Z4+hrcOxcqeOeRAZzH6/9RjOwa2zBs+QEigcImZWUSYlOWkaWhIZ5H679SjTCjKYNHKE36UMKIVDhJgZ75tVwOp9pzjV2OJ3OSISAZW1TWw6XMetswbHsQ3hFA4RdNvsMXQEHb/beszvUkQkAp7fHBoZeP8gG1IChUNETSvIYFpBBs9urPS7FBEZYM45fr2xkgXjcijJTfO7nAGncIiwD8wtZEtlPeVVDX6XIiIDaEtlPfurz3DH3EK/S4kIhUOE3TZnDAkGz2484ncpIjKAnt1YSXJiArcMwvkGUDhE3Mj0VK69LJ/nNh0hGHR+lyMiA6C1PcgLW46ydPooMlKT/C4nIhQOUXDH3CKO1TezZv8pv0sRkQHwxp4q6pra+MDcIr9LiRiFQxQsnT6K9JREfq2hJZFB4dmNleSNSOGayXl+lxIxCocoSE0K8L5ZBby0/ZguAiQS52rPtPL67ipunzOGxMDg/QgdvD9ZjLljbhFNrR28uE2n0xCJZ89vPkJbh+OvBuleSp36HQ5mFjCzTWb2O+/+eDNba2ZlZvaUmSV77Sne/XJv+biw53jAa99jZjf2t6ZYdOW4bCbkDefJdYf9LkVE+sg5x5PrK5hZmMnlYzL9LieiBmLL4bPArrD73wC+5ZybDNQCd3vtdwO1zrlJwLe8fpjZdGA5cDlwE/B9MwsMQF0xxcxYvqCY0kO1lJ3QMQ8i8WhzRR27jzfw4QUlfpcScf0KBzMrAt4HPOzdN+A64Bmvy2PA7d7tZd59vOVLvP7LgCedcy3OuQNAObCgP3XFqg/MLSIpYDyxrsLvUkSkD55Yd5i05AC3zRl8p8s4X3+3HL4N/DMQ9O7nAnXOuc5Z10qgc2CuEKgA8JbXe/3PtXfzmEEld0QKS6eP5tlNlTS3dfhdjohcgobmNn675RjvnzWGESmJfpcTcX0OBzO7Fahyzm0Ib+6mq7vIsgs95vzXvMfMSs2stLq6+pLqjRXLFxRT19TGyh2amBaJJy9sOcrZtg6WLyj2u5So6M+Ww7uA28zsIPAkoeGkbwNZZtYZq0VA5wUNKoFiAG95JlAT3t7NY/6Cc+4h59x859z8/Pz8fpTun3dNzKM4ZxhPamhJJK48ua6CqaPTmVOc5XcpUdHncHDOPeCcK3LOjSM0ofy6c+4jwBvAB71uK4DnvdsvePfxlr/unHNe+3Jvb6bxwGRgXV/rinUJCcbyK0t4Z/8pDpw843c5ItIL24/Us+1IPR9eUDKoLgV6IZE4zuFfgC+YWTmhOYVHvPZHgFyv/QvA/QDOuR3A08BO4A/Afc65QT0gf+e8IhITjMfXHPK7FBHphV+sOURqUgK3zxmU06HdGpBZFefcm8Cb3u39dLO3kXOuGbizh8f/O/DvA1FLPBiZkcrNMwt4qrSCz99wGcOHwOSWSLyqPdPKc5uOcMfcIjLTBudJ9rqjI6R98reLx9LQ3M5zm3S+JZFY9lRpBS3tQVYsHut3KVGlcPDJ3JJsZhZm8tjqg4SmXkQk1rR3BPn5O4e4akIuU0dn+F1OVCkcfGJmrFg8jrKqRlbv06m8RWLRq7uqOFJ3lhWLx/ldStQpHHx066wCcoYn89PVB/0uRUS68djqgxRmDeP6aSP9LiXqFA4+Sk0K8DcLSnh11wkqapr8LkdEwuw+fpp39p/iY1eNHdSn5u7J0PuJY8xHFpUQMOMnbx/0uxQRCfPoWwdISUzgQ/OHxhHR51M4+Kwgcxi3zR7Dk+sPU9fU6nc5IgKcON3Mc5uO8KEri8kenux3Ob5QOMSAe949gabWDn6hg+JEYsKjbx+gI+j4xNUT/C7FNwqHGDB1dAbvmZLPT94+qLO1ivjsdHMbv1xzmFtmFlCSm+Z3Ob5ROMSIe989kVNnWnlmQ6XfpYgMaU+sPUxDSzv3vnui36X4SuEQIxaOz2F2cRY//tN+OoI6KE7EDy3tHTzy1gGunpTHjMLBfRnQi1E4xAgz41PvnsChU028tP2Y3+WIDEm/2XSEqoYWPvnuoTvX0EnhEENumD6aCfnD+e/Xywlq60Ekqto7gnz/zX3MKMzg6kl5fpfjO4VDDAkkGJ9dMpndxxt0pTiRKHtu0xEOnWric0suGzLXbLgQhUOMuXXWGCbkD+c7r5Vp60EkSto6gvy/18uZUZjBkiF4qozuKBxiTPjWwx+09SASFc9tOsLhGm01hFM4xKBzWw+vautBJNJCWw1lzCzM1FZDGIVDDOrcethzooGXtmvrQSSSntt4hIqas3zu+snaagijcIhRt84aw6SRI/ivV/bQ3hH0uxyRQam5rYNvv7qXWUWZXDdVWw3hFA4xKpBg/PONU9hffYanSiv8LkdkUHps9UGO1jdz/81TtdVwHoVDDLth+ijmj83mW6+Ucaal3e9yRAaVuqZWvvdGOe+Zks/iiTqu4XwKhxhmZjxwyzRONrbw8J8O+F2OyKDy/Tf30dDSzv03T/W7lJikcIhx88Zmc/OM0Ty0ah/VDS1+lyMyKFTWNvHTtw/ywblFTB2d4Xc5MUnhEAe+eOMUmtuDfPvVvX6XIjIo/OfKPZjBF5Ze5ncpMUvhEAcm5I/gY4vG8sS6w+w4Wu93OSJxbf3BGn6z+Sh/f80ECjKH+V1OzFI4xInP33AZ2WnJPPj8DpzTgXEifdERdPzr8zsYk5nKp987tK/XcDEKhziROSyJf7lpKqWHavnN5iN+lyMSl3659hC7jp3my7dOJy050e9yYprCIY58cF4Rs4uz+N8v7qahuc3vckTiyqnGFr65cg+LJ+Zy84zRfpcT8xQOcSQhwfjqbZdzsrGF77xa5nc5InHlmyv30NTawf+67XId8NYLCoc4M7s4i+VXlvDo2wfYWlnndzkicWHN/lM8ub6Cv3vXOCaPSve7nLjQ53Aws2Ize8PMdpnZDjP7rNeeY2avmFmZ9z3bazcz+66ZlZvZVjObG/ZcK7z+ZWa2ov8/1uB2/81TyRuRwj8/s5U2nXdJ5IKa2zq4/9dbKclJ4ws3TPG7nLjRny2HduCfnHPTgEXAfWY2HbgfeM05Nxl4zbsPcDMw2fu6B/gBhMIEeBBYCCwAHuwMFOle5rAk/u32Gew+3sCP/rjP73JEYtq3Xt3LwVNNfP2OmQxLDvhdTtzoczg454455zZ6txuAXUAhsAx4zOv2GHC7d3sZ8DMXsgbIMrMC4EbgFedcjXOuFngFuKmvdQ0VSy8fzftmFfDd18opr2rwuxyRmLStsp4fr9rP8iuLWazrQl+SAZlzMLNxwBXAWmCUc+4YhAIE6DwPbiEQfnrRSq+tp3a5iK+8/3LSUgJ88ZmtOq23yHla2jv44jNbyBuRwgO3TPO7nLjT73AwsxHAr4HPOedOX6hrN23uAu3dvdY9ZlZqZqXV1dWXXuwgk5+ewleXzWDT4Tq+94aGl0TC/efKPew+3sB/3DGTzGFJfpcTd/oVDmaWRCgYHnfOPes1n/CGi/C+V3ntlUBx2MOLgKMXaO/COfeQc26+c25+fn5+f0ofNG6bPYbb54zhu6+XsfFwrd/liMSEt8pO8uM/HeCji0pYMm2U3+XEpf7srWTAI8Au59z/DVv0AtC5x9EK4Pmw9ru8vZYWAfXesNNKYKmZZXsT0Uu9Numlr94+g9EZqXzuyc006roPMsTVnmnln361mYn5w/nSLdP9Lidu9WfL4V3Ax4DrzGyz93UL8HXgBjMrA27w7gO8COwHyoEfA58GcM7VAF8D1ntfX/XapJcyUpP49vI5VNY28ZUXdvhdjohvnHM88Ow2as608p3lV2jvpH7o88lFnHNv0f18AcCSbvo74L4enutR4NG+1iJw5bgcPvPeSXz39XIWjMvhr68svviDRAaZR98+yB92HOeBm6cyozDT73Limo6QHkQ+e/1lXD0pjy8/v53tR3Rqbxla1h+s4T9e3MUN00dxz7UT/C4n7ikcBpFAgvGd5XPIG57Mvb/YQF1Tq98liURFVUMz9z2+kaLsYfzXX8/WuZMGgMJhkMkdkcL3PjKXE6eb+dxTm+kI6toPMri1tgf5zC83cbq5jR98dB4ZqdptdSAoHAahK0qy+cptl/Pmnmr+7fc7/S5HJGKcc3zpuW2sO1DDNz4wi2kFuh70QNHVLgapjywcy76qMzz69gHG5w3nrqvG+V2SyID7/pv7+NWGSv5xyWSWzdGJFQaSwmEQ+9L7pnG4JrR7a3F2Gu+dOvLiDxKJE7/bepRvrtzDsjlj+Pz1k/0uZ9DRsNIg1jlBPa0gg/t+uZFNOoJaBol39p3iC09vYf7YbL7xgVmagI4AhcMgNzwlkUf/9kryRqTwtz9Zz+7jFzr9lUjs21xRxyceW8/YnDQeums+qUk60C0SFA5DwKiMVB7/xEKGJQX46MPrOHDyjN8lifTJ7uOnWfHoOnJHpPCLTywkZ3iy3yUNWgqHIaI4J41ffGIBQef46MNrOXyqye+SRC5JeVUDH3tkHalJCTz+iYWMykj1u6RBTeEwhEwamc7PPr6AM63t3Pmj1ZRXNfpdkkiv7Dhaz1//aA3OwS/uXkhxTprfJQ16CochZkZhJk/es4iOIHzoR++w86jmICS2bTxcy4cfWkNqYgK/uvcqJo9K97ukIUHhMARNHZ3B059cRHJiAssfeoe1+0/5XZJIt/64t5qPPbyW7OHJPH3vVYzPG+53SUOGwmGImpA/gqc/eRV56Sl87JF1/GbTEb9LEvkLv1x7mI//dD3FOWk8/cmrKMrWUFI0KRyGsOKcNJ791GKuKMnic09t5juvlhE6s7qIf4JBx3+8uIv/+dw2rpmcxzOfWqzJZx8oHIa4rLRkfn73Qu6YW8i3Xt3Lpx/fSENzm99lyRBVe6aVjz+2nh+t2s9HF5Xw8F3zGZGiEzn4QeEgJCcm8F93zuZLt0zj5Z0nuO2/39bBchJ1WyrquPX/vcXq8lN87fYZfG3ZDBID+ojyi9a8AGBm/P21E3ji7xdxpqWd27/3Nr9Yc0jDTBJxwaDj4T/t584fvgPAr+49NJXsAAAMlUlEQVS9io8tGqtTYvhM4SB/YcH4HH73j1dz5bgcvvyb7fztT9Zz4nSz32XJIFVZ28TfPLyGf/v9Lq69LI/f/cPVzC7O8rssASxe/zKcP3++Ky0t9buMQSsYdPxi7SH+94u7SEkM8OX3TeOD84r015wMiI6g44l1h/n6S7sB+Nf3T+dOvb8izsw2OOfm96qvwkEuZH91I198ZisbDtVy5bhs/u32mUwZrYOQpO+2H6nnS7/ZzpaKOhZPzOUbH5ilI56jROEgAyoYdPxqQwX/8dJuGpvb+eiisfzDdZPIHZHid2kSR6pON/OtV8t4av1hcoan8OX3TWPZnDHaWogihYNERM2ZVr65cg9Pl1YwLCnAJ6+dwN3XjCctWbsaSs8amtt4aNV+Hv7TAdqDQT6ycCyfv+EyMofpWs/RpnCQiCqvauT//GE3L+88QXZaEh9/13juumocmWn6zy5/VnOmlZ+8fYCfrj5IQ3M7t84q4Is3TmFsrk6B4ReFg0TFxsO1fP+Ncl7dVcWIlEQ+dGUxH1s0lnE6/82QVl7VwM/fOcTTpZU0t3dw0+Wjue+9k5hRmOl3aUOewkGiatex0/zwj/v4/dZjtAcd116Wz98sKOG9U/NJSdRVuoaC5rYOXt11gsfXHOad/adIDiRw25wx3PvuCUwaqR0YYoXCQXxRdbqZJ9ZV8Mt1hzhxuoXMYUncOquAZXMKmTc2m0CCJh4Hk/aOIOsO1PD85qO8uO0YDS3tFGYN4yOLSvjQ/GLtsBCDFA7iq/aOIH8qP8lzG4/w8s7jNLcFyR2ezPXTRnH99FEsmpBDeqrmJ+JR/dk23tl3kld2VvHa7hPUNbWRlhzgphmjueOKIq6amKs/AmLYpYSDdjORAZcYSOC9U0by3ikjaWxp543dVby88wS/33aMp0orSEwwrijJYvHEPK4cl8OckiydXC1GnW5uY9PhOkoP1vBW+Um2VNQRdJCRmsiSaaNYOn0U756Srz3WBqGY2XIws5uA7wAB4GHn3Ncv1F9bDvGnpb2DDYdqeavsJG+Vn2TbkXqcgwSDKaMzmFmYwYzCTKYXZDB5VLp2dYyyuqZW9p5oZMfRenYcPc32I/XsOdFw7nc0qyiLaybncfWkPOaOzSZJJ8WLO3E3rGRmAWAvcANQCawHPuyc29nTYxQO8a+h86/SQ7VsOlzLjqOnqTnTem75yPQUJo0cwdjc4ZTkpDE2N42CzFQKMoeRn56i4YtL1N4RpLqxhWP1zRytO8vhmiYOn2ri4KkzlFed4WRjy7m+eSNSmFGYwRXF2cwfl82c4iyGa+su7sXjsNICoNw5tx/AzJ4ElgE9hoPEv/TUJK69LJ9rL8sHwDnH8dPN7Dx6mvKqRsqqGtlX3cjLO45zKiw0AAIJRu7wZPJGpJCXnkJ2WhLZaclkDksiPTWRjNTQ97SURIYnBxiWHGBYUoCUpACpiQkkJyaQFEggOZBAQoyHTDDoaO0I0tYRpKU99NXc1sHZ1g7OtnXQ1NrBmZZ2GprbaGhu53RzO/VNrdQ2tVHb1MrJxlaqG1qoOdNC8Ly/BfNGJFOSk8Z1U/OZPDKdSSNHMH1Mhi6uIzETDoVARdj9SmChT7WIT8yMgsxhFGQOY8m0UX+xrKG5jYqasxyrP8ux+maO1zdT3dDCycbQ18GTZ6hrauV0c/slv24gwUhMMJICCQQSjECCkWBGIAESLHQ7IQEMI8FCdRqAlynm1R7WROdnsHPu3G1cqN05R9CBwxEMQtA5gs7R4d1u7wjSEXS0e18d53+i90LmsCSyvMAszBrG7KJM8tNTGJ2ZypjMYYzOTKUkJ01bA9KjWHlndPenW5f/EWZ2D3APQElJSaRrkhiSnprE9DFJTB+TccF+7R1BzrR0cNr7K/psWztnWkJ/Xbe0d9Dc1kFzW5DW9iCtHaHv7cEg7R3eh3FHkI7OD+qg8z64Qx/anR/0Qce561y4c/+EPuzDWVh6dL7BzbyAOXc7dD8UQKGQ6gyoxMCfQyspkECKt8WTmhggJSmB1KQAacmdX4mkpyaSnpLEiNREDblJv8VKOFQCxWH3i4Cj53dyzj0EPAShOYfolCbxJDGQQGZagk7lIdJPsbK7wXpgspmNN7NkYDnwgs81iYgMWTGx5eCcazezzwArCe3K+qhzbofPZYmIDFkxEQ4AzrkXgRf9rkNERGJnWElERGKIwkFERLpQOIiISBcKBxER6ULhICIiXcTEiff6wsyqgUN9fHgecHIAyxkoquvSqK5Lo7ouzWCsa6xzLr83HeM2HPrDzEp7e2bCaFJdl0Z1XRrVdWmGel0aVhIRkS4UDiIi0sVQDYeH/C6gB6rr0qiuS6O6Ls2QrmtIzjmIiMiFDdUtBxERuYBBGw5mdqeZ7TCzoJnNP2/ZA2ZWbmZ7zOzGHh4/3szWmlmZmT3lnUp8oGt8ysw2e18HzWxzD/0Omtk2r1/EL5xtZl8xsyNhtd3SQ7+bvHVYbmb3R6Gub5rZbjPbambPmVlWD/2isr4u9vObWYr3Oy733kvjIlVL2GsWm9kbZrbLe/9/tps+7zGz+rDf779Gui7vdS/4e7GQ73rra6uZzY1CTVPC1sNmMzttZp87r09U1peZPWpmVWa2Pawtx8xe8T6HXjGz7B4eu8LrU2ZmKwakINd5hatB9gVMA6YAbwLzw9qnA1uAFGA8sA8IdPP4p4Hl3u0fAp+KcL3/BfxrD8sOAnlRXHdfAf7HRfoEvHU3AUj21un0CNe1FEj0bn8D+IZf66s3Pz/waeCH3u3lwFNR+N0VAHO92+nA3m7qeg/wu2i9n3r7ewFuAV4idKG8RcDaKNcXAI4TOhYg6usLuBaYC2wPa/s/wP3e7fu7e88DOcB+73u2dzu7v/UM2i0H59wu59yebhYtA550zrU45w4A5cCC8A4WuiDwdcAzXtNjwO2RqtV7vb8GnojUa0TAAqDcObffOdcKPElo3UaMc+5l51znRaLXELpioF968/MvI/TegdB7aYl1Xmw6Qpxzx5xzG73bDcAuQtdojwfLgJ+5kDVAlpkVRPH1lwD7nHN9Pbi2X5xzq4Ca85rD30M9fQ7dCLzinKtxztUCrwA39beeQRsOF1AIVITdr6Trf55coC7sg6i7PgPpGuCEc66sh+UOeNnMNnjX0Y6Gz3ib9o/2sCnbm/UYSR8n9Fdmd6Kxvnrz85/r472X6gm9t6LCG8a6AljbzeKrzGyLmb1kZpdHqaSL/V78fk8tp+c/0PxYXwCjnHPHIBT8wMhu+kRkvcXMxX76wsxeBUZ3s+hLzrnne3pYN23n77LVmz690ssaP8yFtxre5Zw7amYjgVfMbLf3V0afXagu4AfA1wj9zF8jNOT18fOfopvH9nvXt96sLzP7EtAOPN7D0wz4+uqu1G7aIvY+ulRmNgL4NfA559zp8xZvJDR00ujNJ/0GmByFsi72e/FzfSUDtwEPdLPYr/XVWxFZb3EdDs656/vwsEqgOOx+EXD0vD4nCW3SJnp/8XXXZ0BqNLNE4A5g3gWe46j3vcrMniM0pNGvD7verjsz+zHwu24W9WY9Dnhd3mTbrcAS5w24dvMcA76+utGbn7+zT6X3e86k67DBgDOzJELB8Lhz7tnzl4eHhXPuRTP7vpnlOecieh6hXvxeIvKe6qWbgY3OuRPnL/BrfXlOmFmBc+6YN8RW1U2fSkLzIp2KCM219stQHFZ6AVju7UkyntBfAOvCO3gfOm8AH/SaVgA9bYn01/XAbudcZXcLzWy4maV33iY0Kbu9u74D5bxx3r/q4fXWA5MttFdXMqFN8hciXNdNwL8AtznnmnroE6311Zuf/wVC7x0IvZde7ynQBoo3p/EIsMs593976DO6c+7DzBYQ+hw4FeG6evN7eQG4y9traRFQ3zmkEgU9br37sb7ChL+HevocWgksNbNsbwh4qdfWP5Gegffri9CHWiXQApwAVoYt+xKhPU32ADeHtb8IjPFuTyAUGuXAr4CUCNX5U+De89rGAC+G1bHF+9pBaHgl0uvu58A2YKv35iw4vy7v/i2E9obZF6W6ygmNrW72vn54fl3RXF/d/fzAVwmFF0Cq994p995LE6Kwjq4mNKSwNWw93QLc2/k+Az7jrZsthCb2F0ehrm5/L+fVZcD3vPW5jbC9DCNcWxqhD/vMsLaory9C4XQMaPM+u+4mNEf1GlDmfc/x+s4HHg577Me991k58HcDUY+OkBYRkS6G4rCSiIhchMJBRES6UDiIiEgXCgcREelC4SAiIl0oHEREpAuFg4iIdKFwEBGRLv4/Uw5YHte+pP0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# decreasing noise on std\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "N=100\n",
    "ratio=0.1\n",
    "num_itr=50\n",
    "std_noise = 0.1\n",
    "\n",
    "f=lambda x: 3*x**2+x**4  # function of f(x)\n",
    "samples = np.random.normal(0,1,N) # x-value of f(x)\n",
    "print('true min: ', np.min(f(samples)))\n",
    "for i in range(num_itr):\n",
    "    # sample and select top 10% (max [-a:]or min [:a])\n",
    "    selected_samples = samples[np.array(f(samples)).argsort()[:int(N*ratio)]]\n",
    "    m=np.mean(selected_samples)\n",
    "    std=np.std(selected_samples)+std_noise*(num_itr-i)/num_itr  # decreasing noise on std\n",
    "    print(std)\n",
    "    # fit the gaussian\n",
    "    samples=np.random.normal(m, std, N)\n",
    "    print(f(selected_samples))\n",
    "\n",
    "'''display the function f(x) '''\n",
    "x=np.linspace(-10,10,N)\n",
    "y=f(x)\n",
    "plt.plot(x,y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Results: the noise on std actually delays the convergence process of CE method and makes it more explorative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
